Ori Goshen - Self Optimizing Agents

0:02 my name is Ori Goshen. I'm the co-founder and CEO of AI21. AI21 is an AI lab based in Tel Aviv. super excited to be here and tell you about our work around self-optimizing agents. And so today we'll speak about you know, kind of these self-evolving AI systems. why it's needed, why it's hard, and and why now. And and I mean, you know, the time is now. I mean, we all see the challenges in the in the industry. We'll speak about it. And of course, my I can I can say personally for myself, my wife is asking for me this feature all the time, and I'm probably still in the early beta version.

0:56 So, so so Yeah, let's So, in in in the past few months, we've seen many of these AI systems actually shift from experimentation to production. we're still very early in this in this journey of AI adoption, but we're seeing still we're we're seeing systems that are now being actually deployed at scale. And this introduces new set of challenges.

1:28 and the most obvious one is to operate economically at the frontier. And what we're hearing from customers again and again is that we have this agent, we're pretty satisfied of how it functions and the overall performance, but we know that if we we would want to apply it for every PR or for every call or for every booking or for every transaction, it will be prohibitively expensive. So, and and and this has been the talk in the last few last couple of months. you know, token maxing is basically over. Now, everybody is speaking about token efficiency.

2:13 and that becomes kind of the the focus of the industry. And the question is really how do we get the best possible performance per token investment? Or how do we, you know, basically best get the best real customer outcome per dollar investment? That's the real question. And and I think we we as an industry become smarter about it. this is what we see a lot of customers aspiring to to achieve basically.

2:44 where you want to make the acceleration in the of the AI usage. You want to see the the outcomes of that AI usage. So, that the the the kind of I don't know if you can see it very clearly, but the black line is essentially an indication of the usage of a certain company in terms of tokens. and the the bars represent the cost. And the cost broken down per model. So, you can see it at one stage this company, in this case Coinbase, were were able to get more control over the cost and have that correlated lines basically disjoint. so, that's where companies are are are looking to to to get themselves.

3:34 And and here's a typical a typical flow of an optimization process. So, you have an agent. you start with several configurations. and we'll speak about the configuration space, which which pretty vast. And then and then after experimentation, you kind of discover the Pareto frontier, the set of operating points that you would want to select a point from, and then you you select an agent candidate, and basically you start evolving the system from there. And in many cases, we see that we have a target operating zone in terms of, you know, cost and and quality, and you're trying to get your assistant you're trying to get your agent performing in in in that in that level.

4:28 that's easier said than done. and if you look at the this is like a a very kind of a caricature of the configuration space, you you have a lot of levers to play with. You obviously you have the model itself and the model weights that can be changed. You have model selections. We're now in a world where there's so many different models that have presenting very different capabilities and performance characteristics. But then you have the harness which which also plays a very important role. And, you know, the harness includes the tool specifications, the prompts, the skills, actually the code, the scaffold that you build around it. There's so many moving parts, and and and that that that represents a challenge. So, let's let's look at an a concrete example.

5:22 and one of them is a around the benchmark called BrowseComp. BrowseComp is basically a deep research benchmark, which is is is very viable because the information the answer it's not an open-ended benchmark that the answer is included inside the the given corpora. so the first thing you do is again map the candidates as I showed earlier. so you would try in this case different models with different configurations in this case in different retrieval configurations. You have like a dense retrieval and a sparse retrieval and another strategy retrieval of a late interaction. And what you see here again empirically on this on this benchmark is basically the the GPT-5 family with the late interaction retrieval and the minimax.

6:20 They're basically represent the the Pareto frontier. You would want to select between these two options. that's the very basic and I think most of the teams are are doing these kinds of mapping before they go to to production. But then there's another dimension. there's there's scaling. You can you can apply different execution strategy for for example, vertical scaling like, you know, the amount of thinking tokens or the amount of iterations you give the loop or the different types of fixed repair loops that you can apply.

7:02 Or you have horizontal scaling when you basically try to generate multiple candidates and then select or merge results from those set of candidates. so if you look at and and this is well known for being best of N, right? You you basically use the same model to generate multiple candidates. you run them in parallel and then you compare and you pick the best one. and what you see here in the same benchmark, you're actually starting to discover new types of result. The yellow line here is representing the minimax which was you know the the cheaper but also lower in terms of quality. And now if if you look at the if you look at the chart, you see that if you horizontally scale you generate multiple candidates and pick the best one, you can actually using the same model get to the same level of quality of the top quality candidate in the in the earlier chart.

8:08 But what you also also also see here is that you know scaling each of these models you get better and better better and better results. For example, the GPT-5 also increased quality in comparison to to cost. So basically you discover more options, right, to run your agents with this. Another strategy is basically operating you know a on an ensemble portfolio of models. So not just generating candidates from the same model, but generating candidates from different models. This could be with adjusted prompts and tool definitions, etc. And then again you can take all of these outputs and combine them into the the most optimal result result. And and again I think this is really in line with the discourse we're seeing today moving from token maxing to model maxing. How do we how are we able to harness more and more goodness from from different types of models?

9:22 And what we actually see again under this benchmark, although it's it's apply it's applicable pretty broadly, is that different types of models they cover different areas of the space. And so, if you are pretty diligent about measuring the covariance and how each model is contributing, you can gain a lot of benefits. So, the the sum of all using all models is greater than using them separately. And if you apply these techniques smartly, you can gain a lot of gain a lot of outcomes. so, for example, basically applying this portfolio strategy, you essentially discover a new Pareto frontier where you have better cost performance mixing these different models in different in different proportions. And this is something that can be learned per per task. It doesn't have to be manually discovered. You actually can learn that these different combinations.

10:35 And what you see here for example is that you're able with that portfolio construction, you are able to get to about 50% less cost for about the same quality if you used the previous state-of-the-art setting. So, so really the same thing applied if you compare not just accuracy with cost, but also accuracy to latency is the same same basic principle.

11:09 So, now the you have this you have a lot of options, but now there's there's another dimension of execution strategies, right? You can chain those different this call to models differently. You can use different escalation strategies. You can prioritize calling one models to another and you can set some different stopping threshold which also impact the the performance.

11:40 So, this is another dimension and here we'll look at another just for the for the sake of diversity, we look at another benchmark this we re-bench benchmark. And here's a work where we we actually see that changing the execution strategy also has an impact on different trade-offs. In this case cost and latency trade-off. So, the simple best of n means you run all the candidates, you let the the last one get the outputs and then and then you pick the best one.

12:18 a more efficient strategy is to do it like a cascading order. You start with the weakest model and you you you go from there and when you reach a stopping point where you're satisfied with the you're confident enough about the results, you stop. so, you so, you're making gains there, but you can also do it in parallel and have that same early stopping strategy applied and in that case you you're not saving on cost, but you're more saving on on latency. You're more optimizing for speed rather than cost.

12:54 and then basically what you show here is that you can play with that trade-off. The same level of quality you can apply different types of execution strategies and gain different trade-offs between cost and latency. So, if you care about speed, you can use one setting. If you care about cost more, you can you can use another one. and one final example on the execution strategy is very much like human and if we take the legal example, right? When you come a legal task, we typically have many, many junior interns doing the busy work, and then it rolls up to a more like an associate that synthesize some thesis around it, and then finally we have the partner who's concluding and making the final decision. So, same thing can be applied in in coding. In this case, we show that if you have a weaker model, just generate many, many rollouts, then a stronger model, and reach those rollouts with more relevant information. And then like a super strong model, like a fable-level model, generate the final patch and making the final decision, you actually get better quality, and with much, much cheaper, like 3x cheaper than just using a frontier model.

14:23 So, all the the the gist here is that all these strategies can be learned. The the search space is is really is huge, right? So, it's it's not a simple problem, and doing it manually, you'll probably miss a lot of optimization opportunities. so, we want to find an optimal and automated way to do it. And and one kind of challenge of doing it manually is, you know, what happens when a new model comes in or when a pricing change or the environment change, the traffic also the the task distribution also changes over time. So, you you need to address all of these, right? That's why basically manually searching the space is is is not is not something practical.

15:18 So, what we what we do in at Air and I know I'm over way over time. As we basically we offer a suite for enterprises like a toolkit that helps them given an agent an existing agent no matter what framework or runtime or what models this agent is operating with. We basically help companies optimize their agents in production aligning with their production traffic so it can continuously evolve and get to the you know best price performance and and according to the customers preferences.

16:00 So so kind of to conclude about why we need an automated way to to optimize these agents. We want to make them efficient. We want to make this you know an efficient process. We want to make it observable so people will be able to see the different trade-offs and select from them and we want this to be future-proof so when a new model comes in or when the you know distribution shifts it's very easy to adjust our agents and make them performant in in production.

16:35 You're more than welcome to check our research in in this area in our blog post we post every few weeks about new discovery around agent optimization. And I thank you very much for your time. Thank you.