↑ back

Personal note - 1

blog

This is the second time this is happening so I’ll make a personal note today.

About a month ago I submitted to vlad’s scaling law challenge. And today I submitted to tinygrads bounty. Both times I thought I had the right answer. Both times the AI misled me into believing my answer was correct.

This is in line with how further away from code I’ve been feeling and consequently a little dumb. I’m working on things and I have no insight into the plumbing. This is a problem.

I’ve decided to tackle only one problem at a time and focus only on RL/post-training explicitly and 1-2 codebase exclusively. No more challenges or hackathons or anything until I don’t gain a deep knowledge of what’s in the codebase.

It’s insane. You think you’re on the right track. With Vlad, I couldn’t exactly prove the scaling laws due to the small dataset and the models convinced me that that was all I could get. I didn’t have the correct isoflops and the models convinced me that the answer was correct. Vlad was kind enough to correct me and then i consequently got the correct curves. Similarly with tinygrad, I truly believed that the answer was on the jit level because it seemed so obvious, why touch the backend when the buffer/ARGS can be easily fixed on the frontend. George in his usual style told me to not waste his time and I’ll be banned from the next time if I submit llm slop.

I felt bad for a while. I blamed myself and my bad decisions for longer. And then I realised something important. I have to stop using AIs to find the solution. Using AIs is like looking at the back of the book for the solution before you’ve solved it yourself. It’s the easiest thing to do. And especially for someone like me, cuz I’m self taught, I have no teacher. No mentor. No contact in the industry. No nothing. So talking to the AI feels like the right thing to do. But it’s actually leading me further from the truth. It’s making my brain hallucinate.

No. That is unacceptable.

The recommendations were sound. The plan was not. AI is a rocket engine for me. But I, me, my mind, needs to know the right direction. Would’ve been nice if I found a good mentor. But until then I need to lower my learning rate to prevent getting stuck in the local minima ever again.

Truth must be data and diagnostics from now. It’s not been too long for me that I can’t go back. My productivity will drop. But that’s ok. Endless lines of code don’t mean shit if the end result is incorrect.