Ion Stoica - The Limits of AI Coding Agents: Two Fundamental Gaps in Agentic Software Engineering

0:02 So we're very excited about the power of AI agents. more particular in particular we started in June 2026 and our interest was driven by this some recent papers from deep mind fun search which demonstrated early on that you can use AI to improve the state-of-the-art performance of algorithms and then was followed by alpha evolve showing that coding agents can be used to improve the sort of for or some you know existing systems and techniques and then open evolve was an open source version of alpha evolve which was released around the same time and obviously we have all these kind of agents coding agents like plot codex and cursor so you know triggered by this kind of events we started to run a special seminar over the summer last summer at Berkeley and where the students system students were using this AI evolutionary agents to apply they apply it to their own system to their own research and in particular we went through 12 different projects here.

1:30 >> Yeah. and the results were quite impressive or better than expected. So these to 12 projects included around seven published work by the students in top conferences in systems or AI included also some work in progress and some publication some some research just published on archive and in 10 out of these 12 cases we got results which are better than STA.

2:05 And not only that but it was very cheap to use you know the best models then Gemini and open and GBT to get the results. So it was quite cheap and quite fast. So this a lot of excitements and also since then and we done a lot of work in the space. in particular we taught two graduate classes courses the previous two semesters we wrote position papers we launched the ADRS blog series where ADRS stands for AIdriven research for systems and of course we like we are from academia so we published and we build a bunch of systems and publish a bunch of papers So everything started with very with great expectations and the expectation was that AI will disrupt the system research the way we know it. So what happened actually over the past year?

3:12 What was our experience? So what was the reality? Well, it is true that AI did improve our research productivity. it's great has been great for research. some providing some ideas of course for speeding up some paper writing. So no question has a very positive impact on our productivity. It did also become a big part of our research agenda like I just mentioned early on.

3:45 However, did it really disrupt the way we do do the research? Did it allow us to build new production systems in a fraction of time we used to do? did it allow us to come with new research through truly new research ideas? Well, the answer to those no not yet. So this talk is about trying to understand why what are what are the limitation we encountered and I would say this is not only our story but the story of many others people we talked with for the past several months.

4:26 Now, if you take a step back, the key goal of an AI coding agent is to develop a software system that meets a certain level of assurance with acceptable levels of risks as fast as possible and as cheaply as possible. here the software assurance is an ongoing process to establish and maintain some justified confidence which is supported by cred credible evidence from running the software that the software system will satisfy some specific claims will have some specific properties and in its deployment environment.

5:08 So fundamentally you need to do that by using the resources at your disposal and the resources come in three forms. It's humans, agents, AI agents and of course compute. So the question is how to achieve our goal by best using these resources. And if you think about this as a framework or a a system then the key question to answer is about what are the bottlenecks and then once you identify the bottlenecks you alleviate them or you remove them if you can and then you are going to speed up the development.

5:49 So let me give start with the motivating examples. So this is work we done recently and the goal here is to synthesize a key value store system from scratch. and you know as everyone knows it's a pretty simple system, pretty simple interface actually where you associate a key with a value and you put that pair key value in the storage and then you can given the key you can retrieve the value and in this particular case it was a pretty simple setting. We we have only one server so no distributed environment although the threaded was the server was multi-threaded. So you have concurrency this. So basically you store the data on the disk and you cach in memory to retrieve it faster and the goal is to maximize the throughput and to evaluate it we use this YCSB which is a well-known benchmark for such workloads. So now in the process of using this evolutionary AI agents to synthesize the solution we got one solution at some point which was six times faster than all the previous solutions. A huge improvement. So why was that right? Of course you are curious to understand what led to such a big improvement. Well, it turns out that these benchmarks generated values in a predictable way.

7:21 So when you generate the value in the benchmark, you know, you need to to say what the value it is. And in this particular case, it generated the value by hashing from the key associated with that value and a seed. Now the agent found this out. So, and instead of storing the values, it basically computed the values using this predictive function on the fly. Which meant that because you don't need to store up the values, you only store need to store up only the keys, you can store a lot of more items in memory. So, the retrieve is going to be much faster. Hence the improvement we've seen. Now everyone knows this is kind of you know reward hacking. So if you look about this reward hacking you see it generate is actually is due to two gaps and this is the main topics of my talks. One go gap we call it requirement gap. So you see in this case the agent in the specification was required to return a value which was put associated with a particular key. However, the user intent, the stakeholders we are going to call intent was to store that value, right? Not only to return that value. So here is a gap and then there is a model gap in this particular based on the workload in the development versus deployment environment in the benchmark in the development environment contains predictable values. However, in the real world deployment you have arbitrary values. So here is another gap. So now how is a solution typically working in this case? So let's just take a step back here. So you have a bunch of requirements which basically describe the system needs to be built performance goals and so forth. And you have a model which tries to abstract away the deployment environment. And this is is about where in the environment you are going to detarget the deployment the workload which you think is representative of the real workload kind of the failures and things like that.

9:32 And now a developer or an AI agents take this requirements and the model and generates a program and then this program is evaluated right and evaluated you can have unit test end to end test integration test or you can have contracts or formal method. Now if it passes this evaluation if it is a evaluation sorry it doesn't pass then you provide the feedback to the developer and to the AI agents and then you are going to continue in a loop until the program passes the evaluation and then you are going to deploy it.

10:08 The problem is again that the requirements and the models are an abstractions on one hand of the user intent what the user wanted the system to do and the real world where the system is going to be deployed and these are the delta between those are the requirement gaps you are talking about. So more being a little bit more specific, the requirement gap is there because the intent of the users is broader than the requirement. And here are some examples. There are omissions, things that the system should do but is not specified like store arbitrary client values. There are exclusion like system things system should not do and are not specified like never expose customer data. tradeoffs like for instance a search has a 200 millisecond SLA if is this SLA is exceed if this deadline is exceeded should you wait should it should you return partial results or you should return a failure and there are conflicts which are not specified how to be solved like for instance you want personalization but this chat history but privacy rules prohibit retaining the chart history how you are going to resolve this trade-off he has to be specified now the model gap It's a it's the same you have for the same reason because the re real world is broader than the model the model gap ex so examples are the workload predictable for instance the workloads have predictable benchmarks during the development versus arbitrary development arbitrary values during the deployment you have failures like fail stop test in during the testing and bizantine pulse during the deployment you have dependencies like setable mock versus changing the APIs in the physical in in the deployment case and or you can have adversarian drift new attacks or workloads which happens after release.

12:08 So these two gaps are responsible for the reward hacking and the reward hacking is arguably responsible for many of the things you have seen recently in the news like these agents escaping their environments and going route. it's also they are you can trace back them to alignments and or misalignments and hallucinations and a key challenge here is about under representation. so and you know because you can only judge from available evidence and the evidence is often incomplete. New evidence might invalidate acceptance of a program P during deployment. Examples are implementation P developed.

12:57 It's an implementation developed in and you know under the model M which assume first of failures then you deploy this in practice and as long as you have only face of failures in the deployment environment you are good. the stakeholder accept the implementation but once you get a failure like a bizantine failure like a corrupted memory that's no longer acceptable now why is closing these requirements g so you know gaps are hard so first of all intent might be t tacit contested or changing fundamentally you don't know what you don't know right you cannot think about all the corner or future cases until you see them Right. And this has been noticed for the past decades in many domains. Software engineering, AI, economics with contracts, politics and so forth. and you can also think about this like from a pure solution perspective. If you think about how you can close them, well, one way is to ask people whenever there is many material change on the deployment, you can ask people is this good or not? or you can create an oracle a model or a simulator that predicts every material judgment of the st of the stakeholder.

14:17 the same is true is a model. the model. If you really think about you want a good model, you need to prove that every omitted aspect in from the real world in the model, what what the model is misses is irrelevant for ac access acceptability of that implementation and you keep the claim and the other one this has to the claims have to be valid as the as the world changes.

14:50 Okay. And again has been a lot of work here over the past decades pointing out to this point and this kind of fundamental gap between the model using development and the real world. Okay. So again here to illustrate why it's hard you can ask people for instance to every time on is a changing environment to say okay is this change materially important for the model you are going to have or you can predict you can simulate the real world right but obviously all this direct giving the user direct access and to or complete having complete simulations are not feasible actually even if you have the direct access it's it's On one hand, the cost and latency can be prohibitive. But on the other hand, it's like you cannot you only can reason for the existing evidence. You cannot reason for things you have never seen it, you haven't thought about. And obviously for the model, for the for the deployment, for the real world, you can't have a complete simulation. You can seem to have a simulation. But if once you have the simulation, that's another model. So you are back in square one because you need to demonstrate that the simulation there is no gap between that simulation and and the real world. Now the other thing I want to make and is that now why we are talking about this because we are we we have been you know like like I said these gaps are there and have been there for decades people identify them recognize them. Well, one reason we are emphasizing this because agents amplify the gaps and these are for several reason is context asymmetry. A developer typically has more local and system and organizational context about the task than the agent. The agent which brings broader context but has less specific context about that task. The other one which is very important AI iterates much faster under a fixed requirements model and a fixed evaluator which means they discover and exploit exploitations much faster. Actually you know the developers typically give up or they are not you know to find any of this exploitation even if they want but the model because AI agent is much faster they are going to find it what humans will never find. and is deployment scale especially if you are going now to have an automatic deployment of this of this u programs in production let me skip now I want to say about there is a third gap and I'll and which I want to mention is is and this is between the evaluator and the requirement on one side and requirement and model on the other side so the question this gaps you know It's illustrated by illustrate this gap is that you have an evaluator. So by by passing this evaluator does it guarantee that the implementation is going to satisfy our requirements under model right in all possible executions. And in general the answer is no because for instance you you use kind of unit test and so forth. So it's only a subset of tests you can points you can show that for these points for these data points for these values you satisfy the requirements under RAM but actually there are ways you can do it and you close the gap and this is formal methods and that's why we are not focusing as much on this gap right now however the reason I'm mentioning this gap is because the using the formal specs actually can widen require requirements and model gaps because the formal specs are going to be less expressive say than kind of natural language written requirements and the models.

18:55 so finally you know like like I will say you know does narrowing these gaps help? yes most actually most successes reported so far are in domain with narrower gaps and there are significant work ongoing work to narrow gaps in other domains and here are some examples formal mathematics. In formal modmatics model gaps is closed because there you have formal sematics you have axiomatic systems which fully defined the the word in which the properties have to be spec you know are going to it needs to be true the theorem has to be true requirement gap still remains because informal to formal translation still remains capturing what is your intent in a in a in a in a theorem and so forth hardware systems you know for hardware system you have instruction set architecture you have RTL which bounds the relevant behavior of course it doesn't close a gap to physical effects which are remaining outside the model strong verification also may close the evaluation gap system optimization we are talking about system optimization versus system synthesis with system optimization you are going to be anchored in a code and in this code is going to capture the interfaces HK ideas functionality. So it's kind of narrowing model and requirement gaps as compared with the system synthesis who lacks this anchor and must infer more from incomplete requirements and models.

20:30 And finally world models right what models try to achieve is to narrow the model gap by learning the models part of the model by interacting with the deployment work. Okay. So two more minutes and I'll be done. so you know so how can you close this gap in general? Well to close this gap in general you need another loop which actually includes the intent the user intent and the real work. And this is the first step here is to have to observe and try to to to detect what are kind of the issues unexpected issues misbehaviors. Then you need to diagnose them. Then you need to revive and you are going to revise the requirements. Hopefully improve some requirements, improve the models, improve the evaluators which should reduce the gaps and you do this until you are going to achieve you are assured your level of assurance and risks and let me give back because I'm so you know skip a few slides. so we've done quite a bit of work in this area and but of course much more remains to be done and here are some pieces of work we've done for instance for the loop for development loop evolutionary loop for AI agents for reducing the evaluation gap for diagnosing you know tracing the intent of and from the traces which represents a user interaction with the systems or the agent interaction with the real world and obviously directly trying to to reduce the to narrow the requirement the model gaps.

22:22 So what are the takeaways? So here the takeaways from this presentation. First, the model and requirement gaps are fundamental and cannot be certified as closed in an open and changing environment. Again, because you don't know what you don't know. So therefore, you can only narrow them. The second one is like the humans will remain in the assurance loop because they are needed at least to narrow the requirement gap because they own the intent and likely because they are going to be in this assurance loop they are going to be the bottleneck and the evaluation gap can be closed using formal method but the cost can be increasing the model and requirement gaps because the lack or the limited expressivity of the specification which are needed by the formal method.

23:19 Finally, these are not new problems like I mentioned. However, AI agent exacerbates them because of the context asymmetry, speed and scale. They are going to find and exploit this kind of gaps much better, much faster than humans. So therefore what what do we need to do? We need to use the best practices in software engineering and security and to do to we need to be more cautious and meticulous than ever before and of course do a lot of more and interesting research to close these gaps. Thank you.