Transcript
0:00 Yeah, okay. today I'm going to be talking to you all a little bit about the frontier directions that we're all thinking about in terms of open source agentic RL. what basically sketching what the road map might be towards training something that looks like an 03 model. What does it take to get something that is like a very good general purpose agent for doing long horizon tasks? and we want a thing that can go off and browse the web and write code and look at images and do all this stuff in service of solving increasingly important problems.
0:38 and so like we know it's doable because OpenAI has done it. Other people have done things like this, but there really isn't currently an open source model that behaves in the way that these models do. We have things like R1 which are great at kind of the single turn math and code reasoning problems, but they are not in the general purpose agents world yet. and so I think there's a lot of things that we need to solve along the way, but I think these problems are things that are solvable and that we also kind of like have a a sketch of how to do it. And so I'm going to in this talk be mapping out like what are these questions, what are the roadblocks, and what are like the plausible things we have to do that will actually unlock this.
1:17 And so what we want is models that can go off and browse the web and write code and look at images and do all this stuff. So, this is one of the reasons like everyone went crazy about the CPs because it's like a standardized way to like give all of them tools, but that's really what it is. You want models that are really good at doing multi-turn tool use. That's kind of the 03 magic is lots and lots and lots of multi-turn tool use with multimodal in input in very general settings beyond just chat and code.
1:48 and whether or not you want to call this like 10-minute AGI, I kind of like the phrase 10-minute AGI for just like how to think about 03 is that anything that you can do as a human in minutes? O3 is usually going to be able to do reasonably well. And that, if you can crack that sailing direction of like pushing the boundary of how long these models can go out and do these things for, that is a possible path towards things that become more reliable.
2:15 And so, O3 you can like give it hard problems and it like does a pretty good job after 15 minutes maybe and like sketching out a pretty good version of the solution. And so, with the stable stakes version of this, which is kind of the area that I've been doing some good source work in, is mostly from our off. So, like rather than just like you train the model to like guess the right answer, you train the model with humans in the loop to be able to use these tools in service of solving some problem that you give it rewards based on the final output of this model after the sequence of tool calls.
2:47 and so, we've kind of moved beyond the single turn RLAIF world of like InstructGPT, ChatGPT sort of things as well as the single turn RLHF world of models like O1 and R1. And we want to like have these things become more agentic, and it seems like the path is to incorporate reinforcement learning into this process. so, the the challenges are like the versions of these models that are quite good can do like many, many tool calls. Like Deep Research can do like 100 tool calls.
3:15 you probably want some kind of intermediate verification where you're not just waiting for the final answer at the end, but you want something like turn-level rewards potentially where you want to be able to ensure that the model is getting credit for the the moves it makes along the way because it's no longer just like one thing it's doing, it's doing 100 things, and the credit assignment problem becomes very tricky as you scale the the length of these tasks.
3:41 you might want to do this thing these four things outside of math or code where verifying whether an answer is right or wrong is not necessarily straightforward. you want to avoid like blowing up your context length. Like if you have a model like reading websites, and if you put all the website in context, and the model is like doing this in RL, this like blows up your contact limit really quickly. If you have it reading lots and lots of websites.
4:03 you want infrastructure for managing the resources of these rollouts in terms of computer environments. So, if you're having models that are like doing data science, you want them to be able to like do data science quickly as part of this training process. and you also ideally want to whether you're doing this centralized or decentralized, move in the direction of like everything being async and reliable. because that is like otherwise you have these efficiency bubbles that like pop up all over your computer structure or training structure. And you ideally want to be able to like hide all of that by having everything going with some lag in parallel.
4:40 and so that's kind of what the the path forward seems to look like. and so this is just a little sketch of some work recently about kind of like working towards this where you kind of can do multi-turn RL by incorporating tool calls in the loop. there's been a big sequence of papers of people doing like many replications of like coding agents and search agents. And people are kind of in the public sphere documenting a lot of their learnings of this process. And I think there's a lot of value that is kind of floating around in scattered papers where we just need to like understand the lessons learned, build from them, and aggregate this into kind of training recipes that can become more scalable by crowdsourcing the answers to some of these questions.
5:23 some of the adaptive agents we're using the the number of tool calls scales automatically. the search space as I said can go up to 100. And we also do see that like for really hard problems, doing more tool calls, searching the web more, gives you a nice scaling curve where you get better answers by putting in more effort by like spending more time browsing the internet essentially. and so this is a thing that we want to have a process for scaling.
5:49 we want technology, we want infrastructure, we want recipes and experiments to push along these trajectories. multimodal is also one of the big unlocks in 03, where I think for a while some people were like speculating like, "Oh, what if you have the model like generate images in its chain of thought reasoning, where everything is like end-to-end multimodal input and output?" And it seems like you don't really need to do that because for a lot of the like multimodal things that are useful for reasoning, it's actually just more efficient and easier to have models use like programmatic tools to manipulate images. So like 03 you can like crop an image and zoom in and rotate. And I think there's a lot of mileage in using this sort of thing, where the ability to work with these images allows you to understand them in different ways by shifting the way that the model has them in its context window to emphasize certain information.
6:43 and so I think just like leaning into thinking of modes of intelligence as tool calls is a very powerful like mental model and practical trick for having models do the things that you want them to do. and once you have these kind of things exist in the tool call format, which I think there are several other examples of, then you start being able to have a way to train the model to do the thing. and so like GeoGuessr for example or like like the model is like zooming in to like look at the license plate.
7:18 There's all sorts of these things that humans do kind of instinctually, for example by like focusing our attention. So what does that look like for an LM? How do you like refocus your attention? The sim- the simple way is just crop the image and zoom in. and so that is one piece of the puzzle we probably do want if we're working towards scaling up these kinds of systems. And we want to we want to replicate those sort of models.
7:43 additionally, the I think people were like excited about multi-agent systems for a while, like like CrewAI sort of thing of like, "Oh, I'm going to put my coder agent, my finance agent in a group chat." And like a lot of these are just kind of silly. They don't actually work very well because these models are not trained to do that. They're not ever trained in a way that actually reinforces those patterns. multi-agent RL is hard. I did 5 years of it in grad school. It's like not easy.
8:10 and to the algorithms don't really even exist for the things you would really want to do. but, there are some kind of shortcuts where you do get very focused versions of multi-agent learning that have good synergies with the scaling paradigms that we're already doing. So, some of these are just like allowing other models to be tool calls. so, let's say you're browsing the internet and you want a model that can like go read 100 websites. You probably don't want all 100 full websites in your context window of the main model. But, what you might want to do is have sub-agents whose only job is to answer a question about a website.
8:44 and once you have this, now you can do things much more efficiently. You can parallelize aspects of the reasoning chains. And especially for things that can be done with like off-the-shelf small models, you don't need to train these end-to-end because they're already like decent at being an LLM. And there's a lot of passive reasoning that can be offloaded to just like any good enough LLM, at which point you now can just train the main model to learn how to prompt these models well, to use them well.
9:08 this is also something that we might want to do with things like video summarization. You probably don't want that all in context for most cases. Navigating the web, it might be the case that like having really large images every second in your context window like blows things up really quickly. but, if you can offload this with like kind of more condensed interfaces to using VLMs as tool calls, then you can have much more logic existing in your reasoning chain within your context window, and you can then scale to much longer chains of tool calls without needing as many tokens as it takes to have an image of the website appear in your contacts window at every single step.
9:45 And so this is going to be a problem especially if you're doing this in an open source centralized way. You like scaling to like millions of contacts is like really really hard. There are I don't think there are real like open source replications of token contact scaling beyond like tiny like academic model sizes. And if you want to do that, type of reasoning you want to kind of use every trick you can. And I think offloading parts of this to sub models is like a very effective one that you can specialize with RL because once the sub routine is solved, you kind of freeze that version of the like helper model as a tool call and incorporate this into the model.
10:26 Another thing on contact management is like these models benefit from thinking a lot. But the thinking takes all the tokens. And so if you're doing multi-step learning, you have all these thinking chains that are all interconnected like and some of these models like for inference, they're not actually keeping those thinking chains around. They're like once they're done with thinking per step, throwing away the thinking tokens and then continuing on. And we kind of need new algorithms for doing this. Like the existing paper people are writing about multi-turn RL are not actually incorporating this and it kind of like breaks all the math where you need to like the recent paper blog post like two days ago from Cognition for their like Kernel Bench AI does this in kind of like a simple form where each step is essentially like a final answer but they allow it to like do many final answers in a row.
11:12 And they have like their one verification system to do this efficiently, you kind of do need to do like true turn level training where you have this big forking process where you need to be able to assign rewards to each one. But sometimes if these are like actually intermediate reasoning steps, you don't have a good verifier off the shelf which is going to need both like new algorithm research as well as the ability to do verification for things like was this a good Google period? Was this a good crop of the edge? Was this thing helpful in service of the final goal? So that's a question that we need to be able to answer efficiently, repeatedly in this kind of training.
11:55 and the most one of the most promising ways I think towards doing this is having the reward models also be able to answer harder questions by themselves being reasoning models. So there's a recent deep sea paper on this sort of thing where you train models to be good like generalist reward models by doing inference feeling where they break down tasks into sub criteria, rubrics, and then they can evaluate these numerically based on following the rubric in reference to the thing that they are scoring.
12:23 And this among other tricks, there's lots of like clever reward reward engineering things you can do to kind of have other measures of similarity or quality with respect to like a ground truth answer. I won't go into all the details there, but the core idea there is if you think of reward models as reasoners who then you can also benefit from inference time scaling to do harder and harder tricks, harder and harder problem solving, step verification, you have a way of giving these rewards throughout the process which you kind of need to do if you want to scale to long time outs.
12:54 And but that also introduces the problem of like okay, now reward modeling might take a minute for validating one turn, which really means you want to push really heavily in the direction of everything being async and like happening later, which might mean your model's policy is being trained on terms that happened 20 minutes ago. Which is kind of like in old school RL days, people did this all the time with like replay buffers and whatnot. You really want both your tool calls and your inference and your reward modeling and your training to all be happening in parallel with whatever as much of a lag as you can tolerate as well as your communication.
13:28 and so that is I think one of the big reasons why I like pushing on a sync RL is really huge not just for centralized training, but for things like scaling over modeling, for things like scaling to goals, scaling resources. and I think OpenAI has done some very cool work showing that this is like a direction you can push. You can go off policy. You can allow your models to be trained on like slightly stale data and it still works because these things are pretty robust. And if you set everything up right, you tune your head parameters, and you ensure that's the freezing process the training process is like stable otherwise, then you have a paradigm which allows you to do all of these things. You can scale onto a goal. You can scale over modeling. You can scale on generality of tasks. and everything still fits in this nice async pipeline where everything is flowing from one to the next thing without like blowing up your print bubbles.
14:25 You can also decentralize your skill acquisition. You can train on many things in parallel and it seems like model merging like works really well. That was one thing that I kind of updated on, which is like you can have a model trained on code and a model trained on math, a model trained on Spanish, and you can literally average the weights and it works. Which on one hand sounds crazy, but on the other hand it's like this is kind of what is being done by the local self-pretraining at smaller scales anyways, and it seems like the updates made to model weights are orthogonal enough for specialized tasks that this has actually like totally happened. Things are nice and linear in most cases. Things are nice and orthogonal. And you can get away with a lot of async updates to models that are then merged even without communication.
15:10 and so Cohere did this very extensively in their latest flagship model Command A. and it seems like we have a lot of ways where you can split up different pieces of the process of learning to do X, Y, or Z, board games, or coding agents, or math. And these can all go off and happen and then come together and merge. and if you can do this, then you can train decentralized models with lots and lots of capabilities because the capabilities themselves can be learned in decentralized models.
15:40 so, what does this mean? Like, it's going to be hard. It's not going to be easy to like put all these things together and make it work make it work well and really train these kinds of models super efficiently. But it seems like it's doable. we kind of know the sorts of problems that we have to solve to do it. we kind of know what things are going to be really annoying and how we can kind of shortcut some of them by like using kind of fallback cheap tricks because these cheap tricks are like unreasonably effective.
16:11 And this does kind of give us a road map and a recipe of what questions we need to answer, what things we have to build, and what we have to accomplish in order to train decentralized M minute agent. Okay, we have a special guest that we're pretty excited about. so, here you go.
16:54 actually, we're going to pause for 1 second and just do another transition, so. This is Mochi. Thank you.