Denis Akhiyarov - Are Small LLMs Ready for Coding Agents?

0:02 Hi everyone. So actually this is a a bit of a personal story. So few months ago I was flying transatlantic and realized that as a I scientist I can be very inefficient without a coding agent while disconnected from internet. So I realized that maybe I should develop my own local coding agent and see if a local LLM running on my MacBook can actually handle a realistic coding agentic tasks.

0:45 oh yeah. there is a link to this repo. So I just open-sourced it recently. and you can also see the presentation there. So >> >> this agent is called Ask Me. And it does one small thing at a time. So it's it's basically designed around three ideas key ideas. How to minimize the context that we pass to a small LLM because LLMs are not really good when you pass them a lot of >> >> context for small language models running locally. And the second is that we do very small actions at a time and immediately verify that action.

1:36 And the third thing is we minimize the reasoning as much as possible. Only use it on recovery loops when things go wrong. So here is an example. you give a small LLM so in this example it's Quant 3.6. and basically, you want to just create a small program. I think of a hello world, compile it, and run it. But, the thing fails because actually it compiled it into a temporary directory, but it needs to execute from another directory. So, things like this are much harder for smaller LLMs to realize.

2:21 So, we have to make very, very small changes and verify them one at a time. So, around this idea, we basically develop a loop. So, Ask Me is a very minimal loop. It's just one file. It used to be 1,000 lines of code. Now, it's like close to 2K lines of code. And so, yeah. Basically, you want to minimize number of big planning steps you take, smaller steps, and only go back when things go wrong.

3:00 So, I evaluated this on few hundred tests. And here, this is just smoke tests with two families of LLMs, Gemma 4 and Qwen. And basically, it works except one Qwen model. This was just a sanity check that we can use it. and then, next is actually running realistic tasks. So, this is actually building features for an app. And what I realized it's that that basically, the this agents can do most of the work.

3:45 but, the issue is that they fail to verify their work and have this full loop, you know, observe what happens, go back and solve it. So, basically it passes most of the tests, but not all tests for developing an app. and it's the same story across both Qwen 3.6 and Gemma 4. And for the record, this was run on open router, not locally, because locally it will take me ages to run this.

4:22 And there is a limited memory on my MacBook. next, so I think >> >> basically this wasn't an example just creating a small agent running locally with minimal harness. And it shows that it can basically work locally with a very small LLM, doesn't need a large codebase like Codex or Claude code.

4:54 and yeah, you can look at results and how it works in this repo. I guess I also want to give some shout out. So, I actually compared it against PyHarness, which is the most minimal harness that people use at the moment. it's much more efficient than Codex or Claude code. And I I also did some comparison with OpenHands.

5:24 And basically this is even smaller than both those harnesses for working with very, very small LLMs locally. So, and it's still work in progress. There is still like a lot of issues, but yeah, I'm happy to receive any feedback, any questions you have. Thank you.