Self-Improving AI Agents: Exo and the Bitter Lesson | Alex Krentsel

0:00 As models keep getting better and better, the big question is what is going to stand the test of time? That is, what kind of a harness design is bitter lesson aligned? And the answer is we have to look at harnesses that have a new set of design goals that are flexible, that are legible, that are editable. Where as the models get better, the harness can get better as well. And to do that, the harness has to be able to edit itself, to specialize to the task at hand, but to do so safely in a way where we can trust that it's not going to break itself or lose its state.

0:31 Yeah. So, I'm excited to present today on our system called Exo. And so, the focus of my talk is going to be how we should think about architecting systems that are able to improve themselves at runtime. This is joint work with an interesting cast of characters. My advisers, Sylvia and Yan, who I work with very closely now. but you'll also notice Martin Casado and Ankard Goyal here. Martin it's like a deep systems person even though he's a VC. and he came and gave a talk at Berkeley on where he was seeing agents going and this was around the time when I was also giving a talk and I posted a talk online on OpenClaw in the internals of how OpenClaw works and that talk got some attention. He saw that I went and shared the slides with him and we got together and started brainstorming about what the future of a system like OpenClaw could be. So we started hacking on this together. So this has kind of been a joint work across industry and academia. So just to set the scene a little bit, I think we're all very well aware, every day there's more and more model releases.

1:30 These are just from the last month alone from September. and each time they come out there's a huge focus on how big or fast or performant each of the new models is. and we see these exponential curves that were clearly on an exponential curve especially now in 2026. But the model is empowered by the system that it's put within. So the model is only as good as the system it's in. And so these models act on the world through agentic systems. Maybe the most common ones we're familiar with are codeex and cloud code. There's many others. But the thread throughout my talk, the open systems question, the research question that drives me is what are the right primitives and principles for building agentic systems that are going to maximally empower our models to be effective in the real world. That's going to be the through line of the talk.

2:20 I have at first a few slides on background of kind of where we are. Now, I know you guys know this here at Arena, but these slides I originally made for kind of a broader audience, but just to set the scene so we're on the same page. This is how I see history a little bit, which is we had a phase zero where we really saw LLMs as next token predictors. And I was graduating in 2019 when this happened. I remember Google's BERT, OpenAI, GP, Open GPT early models, Google had Lambda. These are really next token predictors. and then in 2022 with the launch of Chad GPT, we saw that if we fine-tune these next token predictors, we could get them to act very effectively as assistants.

3:00 We had Chad GPT, Claude, and Gemini. And then there was this middle period that was really interesting of LLM given certain tools and orchestrated upfront by a human to act in a certain series of ways. Kind of almost a model of one action passed to another action passed to another action. and frameworks like Langchain or Autogen Crew AI what I would kind of call scoped agents. It was a very clunky time. and finally in phase three which is the tail end of where we're at now and we're about I think to enter phase 4. We moved to LMS that had tools but that could dynamically decide which tools to use at runtime. And kind of giving them that unconstrained ability to make those decisions is what makes these agents more autonomous than the previous state that we were coming from. And this is cloud code and open claw and other things.

3:51 yeah, so we're here. So at the end of the day, all of these systems when you look at them boil down to LLM calls. and there's like this agentic system loop. And the difference between these systems is what context is provided to the actual LLM. And so again, this will be one of my last kind of backgroundy slides, but you have your transformer inference which predicts one token at a time. But then a large language model is repeatedly sampling repeatedly asking the transformer for the next token over and over taking the previous all the tokens you have and asking for the next one. And then we had assistance which repeatedly sample ask an LLM to give a response and interle it with user messages.

4:34 and then we got our scoped agents like cloud code and codecs and cursor that go and repeatedly use kind of an assistant but also provide tooling and are able to execute tooling and interle kind of observations and actions on the world in with the repeated calls to an assistant. then I think we got to earlier this year a layer of an autonomous agent kind of that owns its environment a little bit more than cloud code. cloud code you drop into a directory and you kind of are working in the directory. OpenClaw lives at a level of like it kind of owns almost a computer or server and it acts on multiple agents. It can spin up multiple agents, multiple threads in parallel. And the emerging new layer that I think is about to be really big.

5:16 We're on the cusp of really big things happening here is a self-improving layer. So, it's yet one more outer loop that allows an autonomous system to kind of go and edit its own policy, the tooling that it has, the environment that it owns, the policy by which it interacts with that environment. and to me, this is kind of like the most powerful loop. which is why it's something I'm passionate about and something that I'm working on.

5:42 yes, learning to improve. Okay, so through the rest of this talk, I'm going to talk a little bit about how agentic systems look today. at least in my vocabulary, the challenges of building these agentic systems. Then I'll present to you our approach called EXO, which is an architecture for self-improving agentic systems. and I'll close with some results on benchmarks and some really fun open world tasks. I'm excited to show you guys. and again, please interrupt at any point, if you want to discuss. So what are agentic systems?

6:19 I think here we all know this is a really important emerging class. But what's really interesting is even when I go talk to experts at different companies in academia, you talk to professors, the vocabulary everyone is using is still all muddled. It's all mixed up. What is an agent? What is a harness? what is an environment? these things mean different things to different people. So I'll present my definition which is to me an agentic system is any system that has an AI reasoning component within it somewhere.

6:44 So kind of really you can think of it as somewhere where there's an LLM call. and yeah most common or frequent example of this is a coding harness. but this kind of layer around the LLM call that supports it the scaffolding we call a harness. And I'm gonna my present my definition here. We can debate this, but to me a harness has two and only two tasks. It is a black box that has a northbound interface that assembles context form an LLM call and it has a southbound interface that acts on and observes some environment. Everything else is a black box. Whatever it does inside is up to the harness. And these are the only two things that a harness kind of has to have abstractly.

7:29 And I'm going to use the word policy to refer to all of the logic that has to do with what's inside of that black box. So anything to do with how you assemble your context and act on an environment. now we're definitely most familiar with coding harnesses, but there's a wide range of domain specific agents that are not just for coding. People are using agents to run business queries, to handle insurance claims, for security, medicine, drug discovery, all these different things. and really when I look around at my friends, it seems like, you know, New York seems to be the like hub of companies that are kind of doing agentic vertical applications over top of some domain. SF seems to be the the hub of like kind of crosscutting core ML or kind of agents development. That's the sense I get.

8:13 but good policy decisions very clearly affect performance of our agents and how well they do at their tasks. >> >> question. >> Yeah, please. >> So, can you go to that slide? So I know there's a lot of debate within like domain specific agents or general agents. what's your take on because a lot of people think like if we keep improving the so-called reasoning capability of AI what you just need to change is the environment tool >> like the tool to communicate between so there's no domain specific agents need it >> and from a system side what's your thought and is it like anything that will change given this different thoughts to impact the system design and also like from a definition side what's your what's your opinion on I actually have some slides on this on my prediction at the very end of my talk. So I will come back to this. But I I do want to call out because it's an excellent question. There's a fundamental question people have about harnesses, which is as our models get better, are our harnesses going to get thicker or thinner? Meaning, are we going to squeeze out more gains with time by adding more kind of custom functionality into the harness, or do we need a really minimal thin harness that gets out of the way of a model? And the smarter a model gets, the more it'll eat into the harness's behavior. I have a thought on this but let me save it till the end but that is the question you're getting at right I think. Yeah. Yeah.

9:34 Okay. now good policy decisions affect the performance of our harness. and I just so that we make this very concrete. What is policy? To me policy would be the tools that the harness provides. Meaning usually if you look inside of a harness it assembles in its prompt to up to the LLM. at the start it'll have some block of like system instructions but then it'll have a list of tools and it'll be like you can call this set of tools. that set of tools is part of the the policy of the design of the harness. The way you assemble context, do you take the last hundred messages or do you take the last 10. More messages means more context, but it also costs more money. the prompting that you choose to put at the beginning, the the way you index skills in memory, all these different things.

10:19 Anything that's kind of inside of that black box, that's static code to me is part of the policy. and I claim, and I think people would agree with this, the right decisions for what this policy looks like is pretty domain specific. So if you poke your head inside of coding agents, you'll notice that they often have some sort of LSP tooling, language server protocol, which goes and as you're working on a codebase kind of builds an abstract syntax tree of your code so that if you want to rename variables or move around a refactor code, it it gives a kind of a graphical view to the agent so it doesn't have to work over text. And so adding tools like that definitely makes the agent more effective at being able to refactor your code. It's very different if you're then working on in bio or in system synthesis or for each of these domains there's kind of particular tooling or particular policy decisions that are effective. So these are my premises.

11:12 now how we get to good policy is changing. before we even had harnesses actually I'm curious if anyone remembers in 2012 did anyone get into like prompt engineering? You remember there's a whole craze for like six months. There were like websites where people would share their prompts and then there were like guides on how to better prompt. I remember trying to generate like an image of a bear. and if you ask for the bear, it'll draw a cartoon bear. But if you say like bear 30 mm high resolution soft fuzz like hyperrealistic, then it actually looks really good. So there's a period of time where we were kind of acting as the machines that were producing prompts. and we were kind of learning how to specialize those prompts. And the way I think about a harness is kind of a system that's its job is to write a dynamic prompt and take action on an environment. But really, it is the box that produces a prompt that goes to the LM.

12:04 but getting these harnesses right, picking the right tools for the domain isn't trivial. It requires domain knowledge. It requires time. And so there's been a lot of excitement in building these agent optimizers that try out different policies that try to optimize an agent. Maybe the most known one was called metah harness from earlier in the year from Stanford. Jeppa does this kind of evolutionary work on prompts specifically not agents al all together. alpha open evolve sky discover a evolve is a bunch of papers that basically say give me an evaluation data set of tasks you want your harness to be good at and it will try different variants of a harness and reason about good ideas to add until kind of that metric goes up.

12:48 So, lots of excitement about having these outer optimization loops optimize our policy. this sounds pretty good, but there are significant challenges with this. And I actually think here I won't have to convince you guys of them as much as I do when I talk to maybe other people. I I'm I'll be curious if you agree or disagree with these, but for one, I think people assume that it's easy to get a data set for the domain you're trying to work on, but actually for many domains out there, you don't have a clean data set that's collected that's able to kind of evaluate how well you are at your task. So, if you're an insurance company, just even getting and collecting a well annotated data set of the right behaviors and the right context to then try to optimize your system on is actually a huge feat. Once you have that data set, it pretty quickly becomes outdated because your environment changes and the documents that you would collect change. and this is another thing that I'm seeing.

13:41 We are increasingly asking agents to do longer and longer tasks. So in a world where each task was 30 seconds, you could go and try 100 tasks on a harness variant and see how it did. But in a world where we're asking agents to spend days or even weeks to do tasks like proving a math theorem or synthesizing a system, building a system end to end, it becomes prohibitively expensive to try a 100 variants on a 100 tasks each to see how they do. So this outer loop optimization I think is problematic.

14:10 And the last thing is in that diagram I had here, you pick some outer optimizer loop. pick an agent builder and there's all of these different papers and different techniques and each of them claim to be better for particular domains and so you're stuck with a fixed outer policy. You can optimize the harness but the outer kind of way by which you're optimizing is fixed. So observing these issues in this work, the question I'm asking is, is there a simpler architecture that gives us the same degree of adaptiveness while alleviating these challenges?

14:42 and the answer as a spoiler is yes, which we can do by turning to the reasoning we already have inside the agent. So the key idea we're going to have is a simple one, which is we want the harness to be able to edit itself at runtime. This is not a crazy idea. in fact, you might have predicted this idea based on what I've said before, but there's a cute reason why this is doable now at this layer in the harness layer where in many other places it's not.

15:19 that takes an action you want an environment and there are observations Y that come out of that environment and the harness gets to see those and keep taking actions until the observations match the reference that that it wants very simple model and in these adaptive control loops in control theory there's an optimizer that sits outside that gets to see the same action and environment result pairs and propose changes to the harness policy to to the actor theta now what's really unique about the harness layer is two things First this property called homiconicity the word isn't important but what is important is that the harness is code and the harness also operates over code. So what that means is the tools that we often give to our coding harnesses are our coding tools to read files, edit files, inspect code, reason over what code should be. and so the harness itself can reason about its own state.

16:16 Why I think this is important, why I'm calling it out is because this is not the case for models. If you think about an ML model, the weights for one of your models are way too big to fit into the context like into the window that the model can take as input. So it's not feasible to ask the model to reason about its own weights. You can ask the model to propose ideas to how to train better, but you cannot actually take the model as input to the model. Whereas in the harness layer, you can take the code of the harness as input to the harness's reasoning because they're in the same language. So I think this is actually a really important thing. and the second is optimizer symmetry, which is that these actions, the way you would edit a harness is in the same domain as the actions that your harness takes already. So a change to the harness is a delta in code. It's an edit to a file.

17:06 And the harness already can edit files. So these two things make it so that we can collapse the optimizer policy inside of the actual core harness policy because that policy can make decisions that act on itself because the harness is code and the policy can edit code. and the optimizer actions are also code edits. Okay, this alone I think isn't a brilliant idea. It's pretty clear you could ask a thing to edit itself. I think it's cool.

17:40 But the real challenge here is a safety challenge. If you have a system that can edit itself, you bring in two new challenges. The first is fate sharing, which is if your optimizer breaks your system, accidentally makes a bad edit. When these were two separate things, in the next iteration, your optimizer could fix the system it just broke. But if you pull the optimizer inside of the agent, if the agent breaks itself, there is nothing left to fix itself. So you have a fundamental issue here. You have fate sharing where if you crash your system, you have also crashed your optimizer.

18:14 And the other is your state, your history, conversation history, any snapshots you took is also unprotected. So if you look at something like Codex or Claude code today, you have a dotcloud directory or a docex directory that has conversation history. And if your edits can delete your history, you can get stuck in situations where you corrupt your policy may not go down, your process may not die, but you corrupt your own agents behavior because the state has been erased and you can't learn from state that's been erased. So just trivially rolling back your entire environment. you have to keep around the learnings of what you did wrong to not go down that path again. So you have to be able to reason more carefully about your your state. You have to separate the state from the actual policy as it's running.

19:01 and so how do we architect our system to allow self-improvement while avoiding these challenges? And this is where EXO's approach comes in. So our design has kind of two key steps. This is what aic systems look like today. I took all these policy levers and I put them inside of this ambiguous blob here. Our proposal is to cleanly partition an agent into actually two layers that have distinct and very important properties. There is a upper layer which is the policy layer which is mutable and static fully evolvable and then there is a mechanism layer that is protected isolated from any changes.

19:45 And then our goal is to put the minimal functionality in the protected layer that enables that gives us our safety guarantees while still exposing as much of the policy behavior up in the actual policy layer for self-editing. and our challenge is then to define which of this functionality should go into a protected layer to provide safety and the policy layer to be able to be evolvable. so here I'm going to take one slide to give some context. I am actually not an AI person. I'm an I'm a systems person.

20:21 I spent my first three years doing networking. My advisers Scott Shanker and I'm a core networking person. and this debate isn't a new debate for people that come from systems. So there are decades of literature on operating systems. Operating systems deal with the exact same problem. We have applications and user space that want to do things and want to be able to change the code. You want to be able to run arbitrary applications, but then there's sets of resources that we want to keep protected and make sure that don't get corrupted. And the way that operating systems deal with this is they have a notion of a kernel that provides particular sets of functionality and protects that functionality while exposing a more safe amount of kind of change up in the user space for the application to write code. So when you write your code, you write Python or something, Python you can express whatever you want in user space. But if you want to edit a file, you go through a file call a a a kernel sys call into the kernel which gives you a handle and an abstraction over a file. You don't actually get to edit memory.

21:22 And the debate that the previous operating systems community has had is what the right layer is for placing this interface. What should be protected and what shouldn't be? And there's a very famous paper in the '90s called Exokernel which made the case that we should put the absolute minimal requirements so just what's needed for secure keeping resources secure down in the kernel and leave everything else up in the user space for use. so and provide libraries for this. And so in this way, the application can choose a different file system or different virtual memory or a different networking stack while still protecting resources.

22:01 and we're going to follow the exact same proposal here and for the same reason. Exo kernel makes this argument because it says this gives you policy flexibility to define whatever file system you want that might be more efficient for your application. I could give more examples if people want to talk about systems more. I love systems. Okay, so here's our concrete split. inspired by the sex kernel design in the protected mechanism in that middle layer that cannot be changed or evolved. We put the state which is conversation history and this is really important annotated with costs. So any resource usage gets annotated in this appendon log of what's happened. Okay.

22:43 environment snapshots. So if you ever want to snapshot your environment, that also lives in the protected layer. artifacts which are things like memories or any files that you want to store that are core to the state. We also keep secrets and finally we have the special guardian mechanism that I'll talk about in a second. Everything else lives up in the evolvable policy layer. So there's tooling external adapters the way by which you assemble your context when you have too many skills how you filter down to the relevant ones compaction. So really much of if you look inside of like openclaw or cloud codes leaked code when it came out which I did or codeex a lot of the logic there is dealing with this like what ends up going in the final message that's sent to the model all of that should live up in the policy so the model so the harness can edit itself at runtime and finally I didn't talk about this too much but we decouple both of these from the environment so this is another shift that I think is happening now but people don't fully realize it or maybe some people do you guys might already know But for the past year and a half, we've thought about agents as I will SSH into my machine and I will do claude-d permissions. If I'm really secure, I'll start a Docker container. I'll give it and then I'll SSH into that and do claude dangerously skip permissions. And this way everything is self-contained.

24:03 The agent runs in an environment. The shift that is starting to happen is I think people are looking at agents as running but operating on a separate environment. And so you can have your agent process running on your machine but spinning up a docker container where any actions it takes happen on that docker container. So this is an important distinction because then you could have kind of one agent that's working in multiple different environments and the sandbox thing happens around the environment where the actions are happening not around the agent itself. this to me makes much more sense and it's the general direction I think that we're headed.

24:35 but >> yeah go ahead. >> Yeah. So regarding the split and perhaps this is related to Ehan's earlier question as model capabilities increase presumably something from the green box is going to start moving to the orange box because we trust the model more do you think there are some code things that we should never move or is there some like understanding of that sort and in the OS conversation that happened I don't know 20 years ago did something like this happen as Yeah. Yeah, that's a great question.

25:11 Okay, so to repeat the question back, you're basically asking as our models get better, is this what is this split that I present to you model agnostic or will this shift over time? If I were to trust my models more, would I shift this? That's that's one thing you're asking, right, Abishek? >> that's correct. Yes, that's one thing I'm asking. >> Yep. and the other thing you're asking is in the OS community, how did they deal with this question 20 years ago? I can answer both of those questions.

25:36 no matter how much you trust your model, unless someone shows me some like alignment research that totally changes my my thinking, I don't think there's ever a state in the future that I can trust that my model will never exfiltrate my secrets or will never delete something I don't want. Precisely because when we ask a model to do something, it's taking a limited specification, our words, of what we want, and it's expanding it into like a very rich space of actions. And just the fact that you're going from a little bit of information to a lot of information means there are assumptions being made in that process. And alignment to me is making those assumptions line up with what I want as the human. but one there's just so many different humans that want different things. Like I don't believe that alignment is just going to solve this. where I'll be able to trust where I can just tell my model don't ever delete my state and it won't ever do that. so for that reason, like I here I've intentionally picked what I think is the minimal set.

26:30 And no matter how smart your model gets to have the guarantees of safety that I want, you need your state to stay protected and just managed entirely outside of the model's control. That's my take. Maybe I'll be proven wrong with time. to answer your question about the operating systems literature, has anyone read the Exo kernel paper here? I'm just curious. No, no systems people. Okay, maybe a couple people. there were also other papers. There's like a micro kernel paper. There's a big debate on this.

27:01 if you'll notice at the end of the day, where did we get to? exo kernel did not win. Our laptops are running the traditional operating system with this kernel. This is why you probably recognize this and not, you know, the one on the right is foreign and I'm telling you about it. the reason that that won out is there was a different trade-off there where the whole benefit of this design was to give policy flexibility but it's a lot of work for a application to go and rewrite the file system or the virtual memory or the network stack or the IPC. So in a world where it was a team of engineers that were doing this where it was humans doing this the additional flexibility granted by having kind of this policy up in the user layer just wasn't worth it wasn't beneficial. The majority of application developers just wanted to develop their app and leave everything else to the kernel. But in a world where we're now shifting to it's not humans writing code, we're scaling up hundreds of times more code being written because it's being written by AI. Suddenly, it becomes worthwhile to expose as much as you can because it's not our effort anymore that costs like like we're trying to expose as many levers to squeeze out performance as we can to customize our systems. And now that this customization is automatic, I think that this more exokernon-l like model will win out for the harness and agent layer. Does that make sense, Abishek?

28:18 >> yeah, that broadly answers the question. Maybe we can have further follow-ups, but we can take that later. >> Yeah, I'm excited to show some results. So I I'll leave time at the end for us to talk more. okay, cool. all right, so two more slides and then I'll get to results. And I'm excited about the results. So now I've kind of presented to you the split that we have in our exo harness. And an agent now if I can define an agent for you an ex harness is actually a logical construct that spans these three layers.

28:52 So an agent an instance of a running agent is a particular set of policy and tooling plus the conversation history that's stored the state that's stored in the state layer plus a particular sandbox. And by extracting out this policy into its own layer, we have an isolated target for safe self evolution that won't hurt either the conversation history, the state or the sandbox environment. and we've actually isolated out and can independently roll back or roll forward the conversation history, the environment, or even I don't have an arrow here, but even the policy. If you make a proposal to a change of policy and you don't like it, you can roll back afterwards and they're all rolled back independently.

29:32 yeah. Okay. So, finally, the last slide of the design to fully collapse the system. If you're curious, how does self-improvement actually work? Once you split these layers up, there's two steps. The first is you have your policy up top. we take the actual code of the policy and we mount it in the sandbox to be readr. And so, in this way, the policy can make decisions to take actions in the sandbox. And those actions can do any tool calls or whatever general coding it wants or it can edit the actual policy that's sitting there right on the sandbox.

30:06 And then in that middle layer in XRS I mentioned this guardian mechanism. Whenever the policy wants to swap itself out for a new policy it'll go and call the guardian which builds that code actually builds the code and replaces the running policy with the new policy. but the guardian actually does something else important. It doesn't just replace with the new policy. It keeps both up in parallel to check that the new policy has some safety properties. At a minimum, it could be that your new policy doesn't crash. But a more complex kind of checker guardian G could check that on some small set of kind of test tasks that the that the new harness isn't just really dumb. And if that passes, the old policy process is thrown away and the new policy process is the new running policy. That's it. This is the mechanism by which you can now do self-editing.

30:52 there's automated roll back and you need this to be outside of the policy because otherwise if the policy dies you have no way to to to recover and we get our self- evolving harness. >> So quick question. Yeah, >> maybe it's a dumb question. No, >> how do you make sure your garden function is not modified by your policy model? because the the policy the code that's mounted in the sandbox the editable code doesn't include the guardian like the actual guardian tooling is provided by this protected layer. The same as there's also a database inside of that exoh harness middle layer the protected layer that holds conversation history. You literally cannot act on it because it's not in the sandbox like it's isolated by >> isolated by >> by by architecture. Yeah >> question. So looks like you're doing a roll back when failure happened. So I've seen other systems do this same thing in a different way like the d6 ar they do this spatial temporal composibility and they can completely extract like the module out. So what's the processing counts between like different divine around this? Yeah, they seemed much more focused on like yeah, they had this like multiple components and make changes to certain components and keep certain parts and isolate others.

32:08 I haven't seen that just to be necessary. I kind of treat this all as one big set of policy and as the policy evolves you can roll it back or forward. I haven't seen the need to kind of split it up into components and per component state. I actually found that deep sea cognance paper like really hard to follow and comprehend. it was like very theory. >> They give like basically a formal proof. Yeah. their approach would work.

32:28 >> Yeah, I didn't find that paper super useful personally or I can think about this more maybe at the end, but I didn't find a big need for a lot of theory around different components independently evolving and rolling back their individual state. I kind of treat this as one component. I find this simpler to reason about and also simpler to code up. Yeah. okay. So, I'll just point out now, this architecture makes it so that edits can't erase your canonical history of snapshots. the policy is only replaced with something that passes the guardian, whatever checks the guardian has. G. I didn't really talk much about this, but we also keep secrets stored in that middle layer.

33:07 This is another big problem. people like you you how do you get Claude to not accidentally excfiltrate your like API key which is sitting in your environment variable or in your like just we actually partition that we keep it out of the environment and intercept any messages as they're leaving the sandbox to swap in the secret so that the model never sees the secret. but we get this ability to fully inspect your kind of running log for looking at your own cost cash use effect etc. and you're able to learn in the environment.

33:39 and you have interpreted learnings. So, and the last thing I'll just say doing it this way, the policy process can edit its own, its own guidance on how to edit itself even. So, if it wants to set up some additional observability to inform its own decisions about how to better change itself in the future, it can do that. Whereas before, when you had an outside agent responsible for kind of editing an inside agent, that outside agent was fixed in its policy. the like reviewer agent or whatever. So by having the same agent be responsible for eding it itself, it can also customize its own policy for how to inspect its behavior to the custom domain it's running in.

34:17 Okay, so we built this this thing is a real it's out there. You can go see it on GitHub. I guess here's 1.5K I think it's at 2K stars now. so the exo harness itself, that architecture split up was about 30,000 lines of code. Actually the majority of that comes from the sandbox support. We really wanted the ability to run not just in Docker containers but off on in Firecracker, Daytona, E2B, Versel sprites, etc. So you can run locally in multiple containers. If there's not enough kind of space on your machine, you can treat your agent as kind of a control system that farms out to environments that are in the cloud.

34:48 That's a whole separate side set of benefits. And then EXO itself, kind of the self-improving architecture, ended up being 13,000 lines of code and 2,000 lines of Rust and 2,000 lines of TypeScript. and it's running in production now at at Brain Trust. So they use it for a number of things internally and a handful of other companies also. but for the evaluation, this is what I've been having a lot of fun with the last six weeks. I took a look at two types of environments. So one is a set of closed benchmarks. The typical way we evaluate things I picked kind of continual learning bench. there's sweet tasks for coding. App world for general tasks. SR regime is an infrastructure set of benchmarks and for each of these I compared kind of a self-improving version of EXO with a non-self-improving version meaning without the ability to edit its own code and after an initial period of seeing 20 or 50 tasks it varies based on the benchmark I let them then both kind of run out on new tasks and report their performance. The other one that I'm really excited about is open world tasks. So with a high level goal, open world operating on the order of hours in a dynamic environment. Has anyone here played Runescape?

36:02 >> A couple people have played Runescape? Okay. Others haven't played Runescape. Have people heard of Runescape? Very fun game. It was one of the key games from my childhood. and I'm going to actually show you a bunch of results from Exo playing Runescape out with other players dynamically. and in particular, I'm actually gonna show you and compare it to codeex and pi, which I think are really fair comparisons because they're kind of state-of-the-art agents that we work with today. so I'm going to start mostly talking about the open worlds bit. I'm going to spend maybe five, six minutes on the remainder of this talk and I'll leave 10ish 12 minutes for us to have discussion.

36:35 >> Sorry, can I? >> Yeah. >> Can you talk a little bit about an actual learning and evolving mechanism for the policy layer? Is it some sort of a basian optimization or gradient desiance or how do you >> no if I had more time a different way for me to talk about this would have been coming from all of the AIdriven discovery research which if you're familiar with alpha evolve or open evolve or sky discover or even Jeepa a handful of these things the way they they work really simply they just say here's a system here's an evaluator and here's an LLM run the evaluator on the system take the logs, give it to the LLM and ask it to improve it, improve the system. No Beijian blah blah, it's remarkably simple, but actually if you let the model especially, and here's the thing, we have to be thinking about our systems not for where models are today, but for where the trends of where models of where they're going and if you believe these models are going to get better and better at making decisions, showing them the design of your system and asking what's the way to improve this actually is an increasingly higher return approach. Yeah, >> thank you.

37:42 >> Yeah. Okay. So, I set up Exo with access to this OSRS SDK which gives you like an MCP interface by which to go and call and take actions in the game. and what's kind of cool here is this is actually on a real open public server. it's a huge map. There's many different skills to level up. There's many different actions to take. and all of Exo, PI and Codex all kind of got access to the same tooling. and I gave it three very high level tasks.

38:12 One was a combat objective, which is to take a brand new account and train it up to maximum combat, to maximize its stats on combat. the other was to obtain as much gold as possible, as much money in the game within a fixed amount of time. and the third was to obtain an item that's kind of relatively hard to make. It takes many hours to kind of get there. and all of this was run with GPT 6.1 soul. and what I see is that EXO consistently very strongly outperforms codecs and pi on their efficiency. So all of the tasks across all three were completed successfully on multiple runs. The difference was largely in how much time and how much money it cost to actually get to the completion of the task. But what's more interesting to me is actually I have this yeah so here like this is the overall view at the time of finishing the task. They all finished the tasks at approximately similar times but EXO was 31 to 62% cheaper on these open world tasks. What's really interesting to me is if we just zoom into one of these the combat task if you look at these curves it doesn't start off cheaper. This doesn't come for free. So if I kind of zoom in on this early time, the very first 30 minutes of gameplay or so, you can see that Exo is actually more expensive than both Codex and Pi as it starts out. But if you look and look at at the logs, if you actually read what it's doing, it'll go and experiment with different techniques in the game and then build tools or changes to its own policy mechanisms, each of those is marked with dots that pretty quickly flatten out the curve. and I can give you some examples of what these are in a second. Like one example is for maybe a deep legal task if you're working on like some legal processing there's a bunch of documents and you need to keep all of that in your context to to remember what you're working on as you're referencing many many documents.

39:53 Whereas in Runescape it's actually fine for you to keep a much shorter context. You probably only need the last couple of steps of actions because it doesn't really matter what you did 20 minutes ago. The policy by which you assemble that context is fixed in codeex. Codex has a set of rules with a maximum context window of a million tokens. And when it reaches 900,000 tokens, it'll go and call compact. And so indeed, you see, you can go even look over here on the decision time on the right. You can see pi and both codecs also have these sharp jumps up and down as the context will filling up and then it compresses and goes back down. Very early on, Exo realizes that it doesn't really need longer history. And so it goes and edits its own internals of how it assembles context to keep only the last couple messages, which keeps its context very short. I know that there I have decision time, but the decision time is directly proportional to how long the context is when it's fed into the call. So that's why I make this claim. but you can see that you the cost is drastically lower for Exo to kind of reach its final goal even though the progress looks about the same across the three agents.

40:54 Okay. then the Yeah, please. >> Yeah. So I'm wondering what's the context management technique that the AXO harness found. Did you did it do this? yeah, it would after it would set itself a goal for the next up to 200 turns and then every 200 turns it would summarize first of all within each 200 turns it would summarize it would only keep I think the last like 10 steps that it took and then after and it would keep around some memory note file where it keep track of what it's doing and then every turn it would summarize that down to like a single sentence of what it tried in that turn and then it would keep a running log of that.

41:33 >> Yeah. Did you specifically instruct the model to be cost aware because normally when all these coding agent to stop they don't care about cost? >> yeah I gave identical prompts to all three which for all it was not explicitly I asked it to be efficient. I did not yeah that that's all I asked it actually. Yeah. >> Yeah. >> Yeah. >> a related question. >> the meta froms which is I guess like the from that you're asking the model to improve itself or whatever is that aware of these sorts of objectives or is it sort of very general pace say improve yourself in whatever way you see fit.

42:13 >> Yeah it's it's the latter it's general purpose. So the metaprompts if you looked inside the initial starting policy is more along the lines of like you're an agent that can edit itself. This is what your code layout looks like if you want to go look at it. Here's a file that describes it. it doesn't say anything Runescape specific or even efficiency specific. >> Angive. >> Yeah. Okay. here's the other thing I'm excited about. Once you've done this once, you can go and take the same exact harness you just asked a special you just specialized and go ask it on a brand new account in a new so at a new point in time, new set of players, go work on the same exact task. And I did this for all three. And to be fair, I let Codex and Pi keep their entire history. So same session for codeex and pi. so same across all three. I just gave them a brand new account and said go try this again from scratch. and we can see that like for some tasks it's about the same actually. And it's because for for like earning coins getting money Exo pretty quickly found the optimal strategy for its abilities. And so there wasn't that much time where it was floundering around.

43:13 There wasn't that much time to learn until it got to the right right strategy. But for combat, for example, a lot of time earlier on in the first run, it was learning how to optimize its own ability to do combat. And so in the second run, it was getting twice as much experience per per dollar, meaning it was twice as efficient at kind of leveling up as it was in the first run. And so the gap actually kind of grows even particularly for tasks where there's things to kind of learn more, where there's more repetition.

43:43 okay. oh yeah and I show this here. You can kind of see what happens. Again, it's to me this is super interesting. But like like stays very flat and there isn't even a period where it's like worse than the other two. And actually in the second run it didn't edit tools at all. It already had all the tools it needed and the context and the policy that worked well for combat training. I have all the good results here. as a good researcher I should tell you it's not all good. in particular, there are classes of tasks that are not very well suited to learning repetitive policy edits. And in Runescape in particular, those are doing quests.

44:19 So, if you're doing something where it's like there's actually no pattern across what you're doing, which in these games is like, you know, for a particular quest, you have to go talk to this person, then go collect some resource, then do this other thing, and it's very specific per quest. There is an overhead to trying to build tools to help you do your thing. And if those tools aren't going to be used in the future, this is actually this hurts you. So this doesn't come for free. It doesn't beat on everything. But for the classes of kind of work where there is repetition, which I think is actually a lot of the work we're going to try to be like as we bring these agents into our kind of office work, there's a lot of repetition what we do. So four classes of work like processing insurance claims or monitoring your infrastructure, checking rollouts, like there are certain patterns of things we do that if we can let the harness specialize, we can really drive down its cost. yeah.

45:07 >> Yeah. It was pretty cool. It kind of reminds me of like neuroplasticity to some extent where it's like you repeat thing things and then your neural connection becomes stronger and that's how like humans can become better. They do and they can generalize for different tasks. what I was curious about was if you also study like the memories on like dreaming in addition to this because I'd imagine like if you're to give like a soda harness the ability to just like look at its past reasoning, look at it past action then dream and reflect and then self-improve >> would that need needing to change to harness? Yeah.

45:40 >> How does that compare to? >> This is a great question. So the question is on if I can repeat it back the like concept of dreaming is you at runtime just do whatever you're going to do but you have some downtime when you can look back on your past traces and think about what would I change to do differently if I were in that scenario again based on the trace yes you will get benefit from that I didn't kind of compare it to that because that would be me implementing like yeah another solution I just let codeex run by default and I'm choosing that as state-of-the-art my response is that what exo what I watched exo do was that when you're doing dreaming you are stuck reflecting on a set of traces that you watched You've basically done a roll out and then you reason about the roll out.

46:15 Whereas Exo is aware of the fact that it has the ability to edit itself and as a result if you actually look at the traces it intentionally decides let me try doing this thing so that I can have some logs of this thing to look at. Like that is a key difference otherwise you just do whatever you think is the best in that moment. You don't have the notion of I will actively push in this direction to measure it to decide whether I should do that.

46:38 >> Yeah. Okay. Yeah. So I know you mentioned about like the repeatness of the things so you can better create a working like hardness change. What about like the reward design that actually in your like evolving loop? Do you need to be very dense or actually some sparness is okay because I'm not super familiar with a game. So >> yeah. Okay. This is a great question. So we're talking about reward density and like reward sparseness. Yeah.

47:05 >> in many like real world tasks like a proof you it's a very sparse reward like you know at the end when the thing is proven you don't know the two weeks on the way there but we're seeing that agents are able to kind of intelligently guide themselves towards the final results by reasoning about how decent does this proof look or how close to I know it's not right but is it close or am I feeling like I'm making progress.

47:25 So I'm designing this explicitly for tasks that are you're right one reason this is a little bit easier in Runescape is there's experience that you can measure. So you have some kind of intermediate guidance. what I will say is I have here some results also on other benchmarks that don't really have an intermediate like reward that are like SRGM is one here which is we bring up a an environment infra environment that's literally running your apps and your like services and Kubernetes. We inject a fault somewhere and we ask an agent to figure out what the fault was. No intermediate reward just at the end it has to fix the fault. And I see the same kind of wins in those environments, too.

48:05 So, I haven't done a deeper study on sparity or or richness of reward, but >> obviously where there's a richer reward, it's easier to selfoptimize, but I'm seeing the same things hold for less dense. Yeah. okay, let me just there's a bunch of other numbers here. More or less, I see this hold across different benchmarks. This does not hold for tasks that aren't related. So when I like ran this on terminal bench, for example, I don't see the same benefits because it's all in the terminal, but the tasks for terminal bench are also different. One of them is predict the next chess move and the other is like like like I don't know, train a model that predicts the blah blah blah.

48:39 They're such different tasks that there isn't a benefit across them. There's actually a cost to reflection. So it's more expensive to run something like Exo on those. I just the gap I see and the thing that I was hoping you guys would affirm is there seems to be a huge gap between the benchmarks people are using to say that their models are good and the classes of actual work that we do because the work we do I don't do work like terminal bench at all like I don't have a 100 unrelated tasks. I have one agent that works across a series of tasks that are actually pretty related.

49:08 So to to me the reason that these gains are available and they're kind of low hanging fruit is because the way we're thinking about evaluating what is working well and what isn't is not aligned with how the real world works. That is kind of where I'm coming from. Okay. all right. >> How do you envision a world where like everyone can have their own personalized harness that optimized towards you know whatever task? Yeah, I suspect that there will be profiles of harnesses because I think I work on you might want to I work on a few different projects at once and I actually might want to have a harness specialized to the particular kind of work I'm doing in that project.

49:46 I also like sometimes use agents to help me manage my like health data and other times I'm using them for coding and other times I'm like planning a date and those are very different profiles. >> Presumably you can create a set of >> totally totally >> like the harness should be smart enough to create a set of hearts. >> Yes. Yes. And to me, that is kind of a a tree of almost branches. if you think of the policy as like a git repo, you could have branches that are kind of specialized to certain areas. all right, I I do have one more slide I really wanted to say because this is maybe the overarching theme of what I want to advocate for.

50:19 As we're doing systems research, we have to be moving towards systems research that is aligned with the bitter lesson. And what I mean by this is we are constantly adding in systems research I think surrounding stuff on top of the latest model. People tried they had like a core model that was doing something and they were like let's add an external model that's a advisor model that then occasionally says something to the inner model to help it. And then what happens 3 months later a new model comes out that's better and it just swallows up all of that additional stuff we built over top. All of those papers from nine months ago that were like let's just have a multi- aent system, they are useless now with the latest model because the latest model just does that thinking on its own. And so rather the case I'd like to make is we have to be doing work where when the model gets better, the benefits of the surrounding system get better as well.

51:09 They have to grow proportionally. And so the goal is to design our systems for intelligence where better models make the system actually more useful. And to do this, you have to intentionally rep prioritize your design goals of your systems because things like flexibility, modifi modif goals of your systems, whereas before they were just kind of nice to have because this is what exposes the system to evolution to the better model itself. how to get there is the question is one possible answer. I have another slide, but actually I will just stop stop there. So, if you want to come check this out, the code is up on, on GitHub at github.com/exohness/exo.

51:48 We have kind of forked a little bit. Exo is now there's a self-improving side, but it's also a a harness, an agent runtime. And so, when you go and see the code, you'll notice that. but, yeah, I'd be very happy to take questions. Thank you all for listening.