Workshop: The Open Agentic Stack: Building the Future of AI Systems with Open Source, Open Standards

0:02 so thanks for having us today. Here we are I'm the former global CTO of AI at the Linux Foundation. I previously led the PyTorch Foundation as the executive director and CTO and I'm a visiting scholar at Columbia University. so before we begin I would like to share some thoughts as we are living through unprecedented times. AI capabilities are advancing at a pace that would have sounded implausible only a few years ago. Models can reason across long contexts, write and execute code, use tools and operate software, and coordinate increasingly complex workflows. We are entering a world where powerful models can solve complex mathematical proofs, generate extensive code bases and power systems that take actions in production networks that have real world impact. So, but exactly at this moment when AI is becoming more capable and more consequential, openness is under genuine pressure. So, the idea of restricting or even banning access to open weight models is no longer a hypothetical, right? it's being actively debated in Washington and elsewhere and we are also hearing proposals that can to attempt to control the frontier through chips, compute, distillation, safety testing and model release processes.

1:26 The safety and security and national security concerns behind these discussions are real and legitimate. They deserve serious technically informed responses. But we must also be equally clear about the risks on the other side. Blunt restrictions can concentrate capability, infrastructure, and decision-making to a very small number of institutions. They can transform artificial intelligence from broadly accessible and general purpose technology into something that the rest of society is only permitted to rent from a few labs. That cannot be the foundation on which we build the future.

2:02 Openness is how we create competition. It is how researchers inspect systems, developers adapt them, students learn, enterprises maintain sovereignty over their data and stack and countries build AI capabilities without being permanently dependent on a single vendor. So openness is also an essential part of safety. It enables reproducible evaluation, transparent interfaces, independent scrutiny, shared security tooling, and the ability to identify and correct problems.

2:37 Sorry. Oh, there's no slides. It's just me rambling. so, but open communities allow more people to test assumptions, discover vulnerabilities, compare approaches, and contribute to solutions. But openness does not mean ungoverned. It does not mean being careless, unsafe, or a free-for-all. Openness means that systems can be inspected, evaluated, adapted, and improved. It means that interfaces are interoperable, implementations are contestable, and no single organization controls every layer or decides who's permitted to participate.

3:14 This becomes even more important as we move into agenti because an agent is not simply a model. An agent is a stack. It includes the model, the harness, the tools it can invoke, the environment in which it acts, guard rails, the reinforcement learning systems through which it improves, the evaluation infrastructure that measures it, and the compute on which the entire systems rely. An agent may depend on APIs, credentials, sandboxes, memory, inference systems, orchestration frameworks, and multiple forms of specialized hardware.

3:47 It may cross organizational, cloud, and geographic boundaries while completing a single task. If any one of these layers becomes closed and vertically controlled, the entire ecosystem becomes less portable, less competitive, and more fragile. That is why open models alone are not enough. We also need open environments so that agentic tasks can be reproduced and shared. We need open reinforcement learning frameworks so that more people can improve models and agents. We need open compute layers so workloads are not captive to one cloud or compute vendor. And we need open standards so all of these components can work together without requiring permission from a central platform. The open agentic stack is therefore not one project. It's not one framework or one model. It is a shared architecture built from open source software, open standards and composable components. And that is the arc of today's disc of today's workshop. We will examine four fun foundational layers of that stack.

4:49 Environments, models, reinforcement, learning and compute. Each talk addresses a different technical challenge, but they share a common principle. The future of AI should be modular rather than monolithic, interoperable rather than captive, and open to participation rather than controlled by permission. This is not simply a philosophical reference. It is an architectural, economic, and strategic requirement for a healthy AI ecosystem. So, thank you for being here and thank you for contributing to that future.

5:20 With that, we will begin with our first speaker, Ben Burtonshaw. So, Ben is from hugging face. Ben will introduce open end an open standard and community hub for building, sharing and running agentic environments. these environments define the tools, APIs, credentials, execution context and tasks through which agents operate or sorry learn and operate. Ben will explain why shared environment specifications and harness protocols are essential for reproducible training, safer execution and collaboration across the open source ecosystem. So with that, please welcome Ben to the stage.

5:59 >> Hi everyone. So as you heard, I'm I'm Ben from Hugging Face and I'm going to talk to you about a project called OpenM and opensource aentic RL environments in general. My opinion, my hypothesis is that RL environments are the best way to democratize AI, the best way to kind of take a stake in AI. mainly because they're pretty easy to build. They're just applications. They're just the kind of things that we do on a daily basis.

6:24 for that reason and everyone can get involved. So I I'll work through this in a pretty kind of high level and and ecosystem kind of perspective and then Daniel's going to kind of go even deeper on RL and and we're going to go quite far on it. But if I miss anything, please come and ask me questions afterwards or contact me. So what is an RL environment? Right. The the simplest way to explain what an RL environment is is that it's a world that an actor acts within or that an agent acts within. The kind of original RL environment would be a chess game where the the board is is the is the world.

7:05 The the pieces are are the state of the the environment. The rules are the the actions or the moves that the that you can make within the game. And the the score the game score is the reward. And in reinforcement learning, we would use that reward to update our agent and and to learn from that. So that's just a game, but let's kind of scale that up now to something a bit more realistic. So we could take software engineering.

7:31 If we say that software engineering is defined as a task that's that's in a in a GitHub issue and the the work is in the GitHub pull request and the the score of that work are the results of the test suite and maybe the CI then we would say that we have an environment that is there for for software engineering and this is how papers like like Swissmith and others have built software engineering environments. We could take this and we could then remove the the work that the from from the pull request. We could remove the code from the pull request and get an agent to generate a code in that same environment and use that test suite to get our reward. And then we have an environment for software engineering that we could use to train a software engineering agent. Now we could carry that archetype across many different tasks, right? Email triage or say project management. And we would need to find the tasks, the reward and the state within those environments and build them into applications.

8:28 And so this is what we're talking about with RL environments. Okay. So what does the ecosystem look like? So to explain this, I kind of like to go back to something called the matter cycle. If you studied machine learning at at school, you you would have encountered it. It's the model, annotate, train, test, evaluate, revise loop and it's not so common nowadays because it's a little bit outdated for modern machine learning and we kind of need to update it a bit. So I like to update it to what I call a capability cycle. So if we look at a capability cycle, we would say that we discover capabilities, we then benchmark them, we represent them in data and then we train models on them. Typically we discover a capability with a harness or or with some kind of prompt, right? We we elicit this this capability out of the model and we can see that okay it is able to do a certain thing maybe not reliably.

9:27 We can then evaluate that capability and we'll typically use another software stack for that and we'll have an evaluation harness that will measure this capability in the model and how frequent how reliable it is and and we can iterate upon that as a community. Next, we'll want to represent that as firstly as a data set and then eventually as an environment. And that's where domain expertise will come into this. And so companies like Merkore will do a great job of building reinforcement learning environments around capabilities for which they'll go out and and speak to and hire domain experts in in the respective domain and and build that environment that encapsulates that capability. partially based on on the the benchmark and but ideally without any form of form of leakage between the two and we'll then have a means that we can train a model on and so that's when the trainers will come in another part of the software stack but also another community right now we'll have labs that are training models using these data sets and environments so what you can see from this kind of capability cycle is that environments are the kind of the means with which that we move capabilities from domain experts into the models that we use. And once the models and and and harnesses have these abilities, then we kind of move on to the next one. And we know that right this capability is is in the weights as it were, right?

10:55 Okay. So that's a really high level view. What you'll you'll take from that cycle is that there are many moving parts, many different communities all playing in on the environment. And therefore, it's a really important part of the stack to be open. Right? This is the means with which that we're communicating capabilities. Like imagine if data sets were in a closed format, it will kind of be a it would be a particularly horrible situation. And that's the the the goal of open.

11:26 So if you look for environments, you'll typically go to papers and GitHub and you'll find that they're very distributed. If you look at the recent Kimmy K3 paper, it came with thousands of environments. The Miniaax paper had I think 10,000 environments. Even the first Deep Seat paper had roughly a thousand environments. And all of these are implemented in different ways. Some of them are shared in different ways. Many of them are sold and they're not something that's readily available in the way that open weight models are.

11:57 it reminds me a lot of of kind of being an MLE back in say 2017 when you would find weights. you'd find implementations for models on GitHub and then you you'd go to Google Drive and you and you'd find the weights from there and and nothing is really exposed and interoperable in the way that it is today with with model checkpoints. So that's the problem and then that problem kind of compounds when you start to look at training. Daniel's going to go into this in more detail, but at a high level a training process is is a is a framework, right? We have a library like TRL or Unsloth that we use to to update our model based on some kind of action that it does. And in this environment, we want to use in this context, we want to use an agent harness to perform the task. So let's say it's the software engineering task. That agent harness is going to do that and interact with this environment, use tools in that environment, and then we're going to update the model. So each of these edges, we have a different layer that that needs to be interfaced, right? and we can't have a harness that expects a certain environment definition and we need to also work with multiple harnesses like pi and codeex and claw code etc. So the challenge kind of compounds the other one of the main mechanisms that we're working on in openm to get this to work are environment hubs. So on Hugging Face, we have about 4,000 environments now that can be pulled in a standard format that you can plug into most major training frameworks. There are other environment hubs like Prime Intellect and there are also more coming and we're collaborating with orgs to share our specs so that if they use if they build environment hubs they'll be able to use a spec such that they can download environments and share them in an easy way.

13:48 So in in code terms like this is what it looks like to use openm we imagine that it would mainly be driven by an agent but you you could it's a bit like docker so you'd say open m in it and you would get a scaffold of a kind of hello world environment that that just gives you a generic score and from that you would just need to write the business logic to for your environment or you could generate the business logic with a with a code agent. You could also import environments from different libraries like verifiers and then you can push, pull and fork environments on the hub so that you can share them or or use other people's and adapt them.

14:28 That's the the base layer of of OpenM. And the the features that we're working on next are to be able to discover environments from hubs and then to be able to validate them. So at the moment the process of finding an environment for your use case is quite difficult but if you could search for environments on the hub you'd be able to pull those and get a range of community environments which is quite nice. We can share with each other and then that creates another challenge. How do I know which environment is better? How do I know which environment is going to help on my use case? And that's where this new validate command comes in where we'll say okay this is a valid spec and also we'll we can train a small model to see if it lifts performance on a given benchmark. So with these two commands we'll have a an autonomous loop for these for integrating environments in post- trainining.

15:22 Okay. So the last block of the talk is then how can I use an RL environment. So the the quickest way to get started would be an evaluation and I'd suggest that kind of most people do this because you can represent a typical workload as an environment and you can see how good a model is. So you could take a model as a as weights or or as a just an API and you could define your environment with a set of tasks and a reward function and run the model over that those tasks and you'd get back an aggregate score of how good a model is your environment, right?

15:58 the the main way or or the core way that we we would work it look at this would be with reinforcement learning. So we would take a model or a policy and we would roll it out over this environment and get a set of actions and then we would calculate rewards on each of those rollouts. We could use an algorithm like GRPO which Daniel I know is gonna come come into further where we would calculate the reward for each of those rollouts and then take a mean across the whole group and update the weights based on that mean and then we'd iterate like that. So if you're interested in more detail on this, I've got a few talks where I go into greater detail on this, but at a high level this is how we' use an an RL environment.

16:40 And then in a the next way would be distillation. So in this setting you wouldn't really be able to distill from from strings like you wouldn't be able to distill from from tokens. You would need log probabilities and we would take a student and a teacher model. We would roll out the student. So we'd get it to generate to perform a task inside an environment and we take the log probabilities of of that student on this task and then we would get a teacher to do the same task or to roll out over the same token sorry. And we take the log probabilities of the teacher on those tokens and then update the weights of the student based on the difference the reverse KL from the student and the teacher. That's one way. We could do this in a few different ways. So we could also take the student and do self distillation where we say okay let's take the best example of the student and compare it to the average or we could give the student some kind of privileged information or or hint. We might get that from the environment or from a data set. Like for example, in the pull request setup, we could say here's the actual solution to this code and we give that to the agent at the beginning and then we update the weights based on the difference between its bad run and its good run.

18:00 Okay, so that's the core of OpenM. That's what we're working on. The the main takeaway is that we're trying to make it so that open environments can be shared. yeah openly across the the hugging face hub but also others. And how do we think that this is going to work? How do we think it's going to be effective? The answer to that is is leaderboards. Like we kind of see environments at the moment existing in a bit of a shadow realm. Like they're they're not really they're not really spoken about that much or or they're not really seen within within the community around model releases. And we think that if people can start to build their own environments and try them out and then we put them into leaderboards and we figure out which ones are are good and which ones are contributing to model success then more people will get involved and and we'll share more of these leaderboards. So that's the talk.

18:55 Thanks Ben. Thanks. Thanks. Hello. Thanks Ben. Hello. Is this working? All right. So, there it's actually working now. Okay. so our next speaker is Daniel Honchen, co-founder of Unsloth AI. Daniel's talk has a wonderful direct title. making open models not suck. open models are advancing rapidly. but anyone who's deployed them in an agentic system has encountered some practical problems for sure. looping behavior, malformed tool calls, unreliable formatting, and failures that compound over wrong run long long running tasks. so Daniel will examine why these problems occur and what developers can do to make models more reliable, efficient, and genuinely competitive with closed systems. So with that, please welcome Daniel to the stage.

19:51 >> Hello. >> Hello. Oh yes. Okay. Hello everyone. you know thanks for coming today. yes we have oh wait the okay wait let me check is the oh okay there we go. Okay it's a bit awkward the computer but okay let's see if it works. If the slides you know fail please tell me. but yeah like hello everyone. I'm Daniel from Unsolve. and great to be here. but today I'm going to be talking about how to make open models not fail not suck and how to make it like much better overall. yeah so for you folks who don't know us we're you know we're we're on we are one of the largest organizations on hugging face and we distribute models to the entire world. okay that number is outdated but we have around 400 million total downloads on hugging face. We distribute the largest models and the smallest models and we're the third largest distributed in the entire world. so we you know we also don't just distribute models so we also fix bugs and open source models. So we collaborate with all the large labs and hardware providers in the entire world to fix issues in the models before you get them. so if you're using open source models on your laptop, on your computers, they're most likely fixed by us in some way. For example, you know, OpenAI's GWSS had some issues. Llama, Jamma, Mistro, all of these models have some issues. And you know, we're here to like, you know, help make the models much better for people to use.

21:17 recently, most people should know, you know, Kim K3 has been launched. it's the you know world's best openw weight model that anyone can download. So they just released, you can download this on hugging face. it's quite large, you know, 1.5 terabytes. but you can download this on hugging face. It was just released just a few days ago on Monday. and you know, for what we did is we quantized it down to one bit for you to use. so it's not, you know, 1.5 terabytes anymore. It's around 600 GB. and Kimik3 was actually 4bit.

21:50 so it's actually not that easy to quantize anymore because, you know, 4bit going to one bit is not that much savings. but we show that you can retain 70 76 to 78% of accuracy but you can reduce the size by 82%. so it doesn't, you know, when you reduce the size of a model via quantization, you're not going to make the model, you know, 82% dumber, it's only like, you know, 16 or something% dumber. And we show, for example, some examples, a one bit KI did pretty well on some prompts. You know, this is one bit, okay, versus, you know, the Kim K3 at 4bit. So the full precision, you know, versus Claude, Opus, and G GBD.

22:29 Deepseek just released an update yesterday. Was it yesterday? Yeah, I think it was yesterday. but yesterday they released, you know, 0731 for Deep Seek V4 Flash. so that was an interesting release at a weird time. and then they just randomly dropped it at some weird time zone. and so we also quantized it down for people to use. so you only need to use 90 GB of u of VRAM or RAM for you to run Deepseek V4 Flash. and it is I would say it's the best model for its size. so Kimiki 3 is still much better, but it's ginormous. but Deepseek V4 flash can actually fit on your local computer and run it. so it works very very well.

23:07 so like you know as open source models and closed source models both of them are getting better and better over time. for folks I'm not sure if you have seen the meter plot. So the meter plot shows the dub the meter plot shows how much progress is there in AI. so like for example you can see claude the plot is a bit outdated because they have not updated it. but you know models can take models can do tasks that take humans 16 hours to do. and it can do this very very well. this is a 50% probability.

23:35 so the 80% probability is a bit less. I think it's like six hours or so. but you can see models are progressing at an exponential speed if you log it. So if you do a linear trend you can see that the exponential progress is much more clear. I also plotted you know mythos or like you know fable as the red line. and also GBD 5.6 6. so this actually removes cheating. so some of these models they like to cheat.

24:00 if you do include the cheating examples, then GBD 5.6 is 127 sorry, 270 hours. so if if a human can do a task that takes 270 hours, GBD 5.6 can also do it. however, that's probably not a good example. So if you remove all this cheating examples, it's around 16ish hours. but you can see over time that models are improving at a dramatic pace. If you look, if you squint a little bit, you know, if you squint a little bit, there is actually this horizontal line area, right? So, if you can see around 2023 to 2024ish, there is actually a plateau approximately if you can squint a little, right? So, GBD4 to GBD40, there was not that much progress. and the main reasoning behind this is because reinforcement learning and reasoning was not invented during that time.

24:51 and so if you plot two different plots like line plots a trend if you just plot you know the non-reasoning time frame versus the reasoning time frame you can see that if we did not invent reasoning so 01 if 01 preview if openai did not like you know tell everyone you can do reasoning then AI models would have plateaued and so if you can see the gray line if you just plot you know if you just fit a trend to the models before the reasoning age all models would have plateaued by now and the doubling time. So if you so every single seven months the models will get double in capability. that was the olden days. but now the green trend is when you add reinforcement learning, you add reasoning and you can see the doubling time has shrunk to 3.5 months. So this means you only need to wait 3.5 months for the model to be double in capability. so you know Opus 5 just got released a few days ago. wait 3.5 months and you'll get even better model than Opus 5. but remember the trick is if the trend continues right so that's the fundamental question will the green line continue and continuously keep increasing for AI capability or we will go we will go back in time and go to the you know the gray line again right will AI just like slow down and slow and then we have to like reinvent something new again so this is a question that we will see in our lifetimes so you know all you need to do is wait a few months and we shall see right so like if you models are plateauing then this means AI has plateaued so all you need to do is wait 3.5 months to 7 months and we shall see the trend so there is another very cool plot by artificial analysis so they show open models versus closed source models and you can see the so the blue line is open source models and the black line is closed source models and if you see very closely open models have very very closely reached closed source models a better a better like indication I you know I'm not you know my favorite gauge if you want to like look at one benchmark you know don't just look at SWE bench or deep SWE or whatever your favorite benchmark is my favorite one is actually weird ML so if you search in Google weird ML this one is my favorite benchmark and the reason why it's actually very hard to game and you can see the blue line is open models and the green line is closed models according to weird ML fable fable 5 is pretty good it's very expensive so I would suggest you to use Opus 5 high which is according to weedml the best for its cost and accuracy. but for open models Kimik3 is extremely powerful. you can see for example they also plot the accuracy across models. so fable 5 okay that has the highest accuracy at max reasoning but it is very expensive. it's ginormously expensive. the next one is Opus 5 max opus 5 high and so on. And you can see Kimik 3 is also on the leaderboard.

27:44 so open models are definitely catching up. In fact, if you squint a little bit at the plot, so this is a bit delayed. If you squint a little bit at the plot, there was actually the delay as I was, you know, talking about 01 preview and open models. and so this delay is what we call the open source drought, which was around 1.5 years ago. and in fact, if you take the closed source models minus the open models, you have this trend. which is the progress of open models. they're closing the gap. this this line had is a bit outdated. so it's more like two months delay. but you can see that you know with the invention of reinforcement learning and Deepseek publishing how to do GRPO with you know DeepSc R1 open models then again went back to trend and so in fact if DeepSc1 did not publish GRPO open models would be 16 months behind closed source models but now it's only two months behind and you know there is this good blog post showing a you know trend extrapolation by 2026 December so in a few months we shall get an open model that is as good as a closed source model. but you know this is all trends so you know who knows if this is going to happen. All we need to do is wait a few months and we shall see.

28:56 and so we shall see if open models can in fact be as good as a closed source model by December 2026. another trend I don't know if you guys saw do you know open AAI decreased prices of GBD Luna by 80% a few days ago. This might sound very shocking but actually it is not. and you can see on the deep suite, I don't know if folks use the deep suite benchmarks, but the deep suite benchmark shows that GBD 5.6 Luna is extremely extremely efficient, cheap, and also very smart. and so, you know, the plot, the deep sweep plot is, you know, to the right, to the top is the best. and so you can see GBD 5.6 Luna has dramatically shifted the, line. and so the main question is token prices are plummeting, but is this reasonable? You know, people are thinking, okay, maybe OpenAI is just shedding all that money. you know why can deepseeek charge you know 10 cents versus you know other providers charging $1, right? Is this actually correct?

29:51 so actually I did a calculation over the before the talk. and so if you I don't know if you folks know seminary analysis they released something called inference X and benchmarks for VLM SG lang and other inference engines on Nvidia GPUs and AMG GPUs. and according to their benchmark DeepSeek V4 Pro so the biggest one on one GPU one B300 B300 GB300 you can do 13k tokens throughput. if you calculate this that means one hour you get 47 million tokens. if you saturate the GPU, 47 million tokens, then if you look at the price of GB300, it's around $5 per hour approximately per GPU according to a semi analysis, right? $5 approximately. and if you calculate this, right, you can do 47 million tokens per hour, you know, $5 divided by that. That means that Deepseek V4 Pro, the minimum floor price is 11 11 cents per 1 million tokens.

30:50 And what does DeepSeek charge? Deepseek charges 44 cents per 1 million tokens and 87 cents per 1 million tokens output. So this means that if you run deepseeek v4 approximately it's not a loss right it's not like open shedding money or like deepseek is shedding money in terms of running these models and that is why openai can reduce the price of GBD 5.6 Luna 220 cents and $120. and why assuming GBD5.6 Luna is the same size as Deepseek V4. Okay, it's most likely not, but it's assuming it's the same size. Then OpenAI still raises some money around $1 per 1 million output.

31:25 if you extract also Kim key3, so Kim K3 is a, you know, a new architecture, it's still not very optimized, but if you look at semi, you know, semi analysis is analysis. Kim K3 takes around 5k tokens throughput per GPU. so it's around 2.6 six times less than deepseync. and if you you know according to Kim K3, I don't know if you guys use the model, it is $15 per 1 million output. that is extremely expensive. And why? Because the if you divide it so it's actually 29.

31:59 The floor price for Kim K3 is 30 cents per 1 million tokens. so Kimi K3 is get so Kimi Moonshot and the other inference providers are getting $14.17.70 on your API cost. so if they wanted to reduce price they can all the way to 30 cents. obviously well you know you still need some revenue and some profit. so the price of these models are still slightly inflated and Deepseek is the only one which is charging you know at close floor prices for their models. you know, another plot that I really like to look at is Arena. So, Arena has this like para efficiency plot. you know, you know, folks probably looked at arena scores and stuff like this. I don't really like the arena score, but the good way is to look at the par efficiency plot that they publish. and you can see that, you know, so to the top to the right is always the best as normal. but you can see there is a lot of models that are in this you know par efficiency line and you know for the performance that you want you should look at which model you want to use and so many open source models are also on the par efficiency front for web development so you know if you want to do HTML development web development and stuff like that you know Opus 5 max for now is the best however Kim K3 is just behind it so if you want to use like a you know a slightly cheaper model and you know if you wanna you to have still very good accuracy. Kimik K3 is a very good model. GLM 5.2 is also on the you know on the plot and other models. but you know if you don't want to pay API cost there is another way you can run the models locally on your computer right and the cost becomes zero right not any cost just electricity cost okay some setup cost some you know front loaded cost. but yeah essentially that is what we do essentially we can reduce a model size to very very small bits like one bit by using dynamic quantization. and you know, we show many, many Twitter posts about this. You know, we did it for GLM, we did it for Kimmy, we did it for many, many, many other models. so if you go to our hugging face page, you know, you can download these models that you can run locally on your computer.

34:01 some of them are very big. So, you know, Kimmy is still 600GB. So, you know, I'm not sure if that can run on your computer. but, you know, we're trying our best to like reduce it to like one bit. and even below one bit, you know, how can we make Kimi like, you know, 250GB, for example. for example, a very good example is GLM 5.2 one bit. you know, you can see this is oneshotted prompt. So, we do not like, you know, tell it to like edit bugs or, you know, fix bugs and stuff like that. And you can see that GLM 5.2 does very very well compared to the other, closed source models. and remember this is one bit versus full precision. so this is not like some full precision model. we also showed you know perplexity benchmarks, you know, to show you that one bit works well. cho divergence. So another way to benchmark these models is to use kale divergence. and yeah so oh wait how much time do I have left or is it oh okay I shall skip this then okay I will skip this. Okay the last thing I'm going to talk about is arc AGI. So you know folks if you know you know a few days ago OpenAI posted that in order to make models good they showed that the ARC AGI benchmark is actually a bit bad for GBD and the reason why is because they had a rolling truncation of 175k tokens and they did not enable compaction and they did not enable preserve thinking and if you just enable these two flags you can make open AI's models increase all the way to 40% accuracy you don't need to do anything just enable compaction and do preserve thinking and you will get high accuracy. and I was going to say like other stuff like oh whatever there's not enough time we have some slides I'll push some slides off but yeah thank you for listening I guess. yeah thanks Daniel. All right, so next up we have Shang Yang from Radics Arc introducing Miles, a flexible and scalable reinforcement learning framework for enterprise facing agent training.

36:06 >> okay, good afternoon everyone. thanks for the invitation. my name is Shan and I'm a research resident at RAD and a PhD student at MIT. So today I'm very glad to introduce mouse to you guys on behalf of the team and to begin with mouse is a production facing system for reinforcement learning frontier language models for with very fast speed and stability and today we will first give you a very brief overview of the overall architecture of mouse and then highlight some recent new release and new features of the mouse systems. So let's dive in and first of all the first question is what is mouse and it is an open source IO engine for frontier models developed by radxark and the key the key point for mouse is that it's a stable efficient and reproducible reinforcement learning system at scale.

37:01 So you can minimize the effort to run a reinforcement learning job on you know you know modern language models with different kind of environments and different kind of reward designs. So at a high level there are three layers in mouse. first of all is we have a SG as a reality engine which is specialized for high throughput monitor generation and it's also very compatible with nowadays agentic agentic system agentic environments and agentic info and in the middle we have the orchestration layer which organize everything like a robot engine and backend training systems as as a whole so you can makes our system run very efficiently and finally we have different for different kind of training engines such as Nvidia action and FSTP.

37:52 So it's also very flexible and you can customize your own training engine on your like own computer computer resource and here are a key picture of the mouse architecture. So on the left we have robot engine which generate trajectories by interacting for example with the external environment or an agent framework and SGL provides a generation layer. So while the environment itself can be very customized or alternated based on the customer's you know requirements and the in the middle the system turns the trajectories into some rewards some loss functions etc. So you have the training signals and finally we have the as we mentioned we have the maxron or fstp as the training back end. So you can update doing the forward and backward and computes the loss and outplace of model weights and finally this model weights will be synced back to the inference engines the ro engine so the whole pipeline can run in the circle and why we use as a raw back end there are some just some advantages first of all it's co-evolved with a workload so including the partial raw support and we can have dedicated ro control. So it's very easy to have your workload specified and second it's raw designs it's router design is very high performance. for example you can have a very well-designed cache well load balancing for high group generation and it's also natively supports asynchronous generation with training engines which is a very important feature in nowadays reinforcement learning algorithms and it's also fast moving and fast moving and there are a lot of optimizations for examples as SGA support speculative decoding with mouse so you can use speculative decoding during the real stage of reinforcement learning And here are some key points that why mouse can change IO develops workflow.

40:02 So as we mentioned before the first thing is that it's customizable. So we can support different kind of workloads just with some customizable function pass. So it can be very easy to define your own IO workloads and your own like reward function and some update method and it's also modularized and modularized and non-invasive. So it can be very easy to change to different training back ends for example as we mentioned FSTP and max and we will also support different kind of chaining back ends in the near future.

40:37 And here are some more technical details about the system design. So first of all we have the route engine and we support multi-turn agentic route as we mentioned a lot of times before and specifically we support token in token out tit for short which is that which means that we don't need to do the dec tokenize and detokenize again and again for the same multi-turn generation so it can save a lot of cost and also makes the training more stable and we also have R3 fore models which means route routing replay so when training ARO training MOE models with ARO so you don't have to worry about like the replay kind of things which means that for different runs you have different router a different expert routed for the same for the same request we don't want that so we have the R3 to make sure that it's fully reproducible and on the training side in we also support low precision RL such as end to end MX FP8 support and we also support them per token and VIP for quantization for the robot stage so that it can greatly decrease the Chinese decrease the Chinese latency and increase the Chinese throughput and all these kind of things has been verified in our latest block with the real world training workloads and we also have the lower support for resource constraint scenarios so that you can train large model with a smaller number of GPUs and here is the current support matrix of the mouse. we recently launched the day zero report day zero support for newest models such as Kim K3 the inkling from syncing machines lab and we also support like deepse v4 non ultra etc and we also have a wide range of precisions as we mentioned and also wide range of hardwares and more importantly we support different kind of fetuning recipes not only SFT RL we also support OPD on policy distillation And we have verified the mouse framework with over 1,000 GPUs for IIO training workload. And we accelerate the Rio training throughput. Like for example, for Kim K 2.6, we can speed ups the generation throughput to one to 12 to 12.5K output tokens per minute per GPU. And we also have optimized the stability such as the R3 tit as we mentioned and the precision align training which help us to decrease the training inference mismatch and we also supports for tolerance recovery and we support different kind of system like cluster back ends such as array on kubernetics slurm and bare metal etc.

43:25 and finally I'd like to mention some of the our recent releases like day zero support for Kimik K3 which is nearly a three trillion parameters model and we do the finetuning for with mouse on like different kind of tasks such as the am mass problem. So we through the a fetuning we can increase the accuracy from 40ish to 7 76%. And for the inkling model from syncing machines lab which is also a one trillion level parameter model we use a full parameter finetuning and the Laura finetuning and we can we also optimize the Laura adapter sync synchronization which means merge the Laura rank into the SGA inference engine. So this kind of things we optimize it from almost 50 seconds to less than three seconds.

44:19 And finally we have a recent blog post about mixed precision training as we mentioned before. So we verified that with different kind of precision choices such as MX F8 or MVIP4 we can maintain accur models accuracy and stability during RIO training and we can also fundamentally and accelerate the road speed and Chinese speed by a great margin. Yeah, I think that's all about today's presentation. Thank you very much for patience.

44:55 Thank you, Sean. All right, our final speaker is Ramal Bardage from he's a co-founder at of Sky Pilot. So, Ramal will make the case that AI needs an open compute layer. please welcome Ramal to the stage.

45:32 All right, there you go. cool. Thanks everyone. I'm Romel, one of the co-founders of the SkyPower project and today, you know, we heard really good talks about open environments, how to do RL in in an open manner. I want to make the case that there's a need for a open compute layer for AI and specifically something that sits between your workloads and the actual GPUs where your workloads run. So building AI infra is really hard right and let me walk you through it from OpenAI's perspective. You know back in 2016 they signed an exclusive agreement with Azure to have Azure as exclusive provider for the GPUs. but in 2024 they signed yet another deal with Oracle for $10 billion to get more more compute and only just a year later they signed another 11 billion deal with core to get even more GPUs right and even that's not enough like you can see Sam Olen tweeting about like hey we need more and more GPUs just give us a call if you can like secure more GPUs for us so if the most capitalized company on the planet right cannot secure enough GPUs and if they're not able to get it in a single cluster there's a good chance you also probably cannot do that right now and so this GPU crunch is very very real and if you look at the typical AI organization today you know it typically you have a AI team which has say some neocloud reservation maybe they signed like a two-year or threeear deal with one of the neoclouds when hyperscalers didn't have capacity then they added some on demand instances maybe from the hyperscalers just to supplement that capacity but then they also end up having some kind of on-prem clusters maybe for regulatory reasons or they just want to keep data onrem and then you have customers in Europe so you need to spin up clusters in Europe as well so you need to maintain another cluster there and then finally you need a ton of CPUs to run your environments and sandboxes and for that you need to maintain another CPU cluster so each of these is sort of a decision made at a point in time where it made sense to do that. But now you end up with this like big collection of clusters that you're now responsible for managing, right?

47:49 And in some sense reinforcement learning makes it worse, right? it makes it from it it makes it a scheduling problem because if you think about reinforcement learning today, it's really it is this whole thing is one job but you can see it's composed of many subcomponents. So for example you can see the trainer itself needs a lot of VRAM needs many beefy GPUs but at the same time the rollout server the one which is actually generating the trajectories needs to doesn't need as much VRAM but needs a ton of them so that it can like generate lots of trajectories and then finally to actually execute for example if you're doing like code generation models your sandboxes need like a ton of CPUs in parallel to run all the trajectories and you also and need very fast cold shots on them.

48:39 So the question then becomes like why not just use existing solutions right like cluster management has been we we've been doing it for a long time so why not just use slur kubernetes right slowmo is pretty good I mean built two decades ago from the HPC world and really you know built for advanced scheduling capabilities one of the best I would say you know kota and priority management systems out there u but it really doesn't support isolation in the form of containers yes you can bolt on like pyroot and more container you know layers on top of slum but it wasn't really designed for it and it wasn't really ever built for serving so it was more for like batch jobs.

49:20 Kubernetes on the other hand came on the scene about 10 years ago. really popular for microservices you know and it's very very elastic and sort of built for the cloud in some ways but has like a super steep learning curve. Like if you hand Kubernetes to a researcher they'll be like how do I work with this? They'll just probably just have to go to their agent to make it work. and it also doesn't have lot of primitives that are needed for AI specifically around like gang scheduling. Again those are bolt on through like schedulers like volcano and so on. but it's really hard to make Kubernetes work for AI but more importantly both are like single cluster managers so they still don't pro solve the problem of like hey I have like 10 different clusters running how do I manage them right so what's really missing is not a betteruler but something to manage all these different schedulers and platforms that we have today right so the goal is not to replace Kubernetes they are really good at what they do but rather you want to bridge this gap where you have some workloads like you know training serving sandboxes that you want to run on your compute which is fragmented across like kubernetes sl and so on and so the idea is if we can have open unified compute layer right which can optimize availability figure out what's the best place to run our jobs handle their execution and so on and do this in a way where your workloads don't need to sort of know what's under the hood that that would add like a lot of value to the ecosystem And that's exactly what we've been building at Skypawit. You know, it's a completely open source platform to use AI compute wherever wherever you have it. And the idea is you bring your own framework, right? So you bring Ray, PyTorch, Openn, Unsllo, your your own frameworks, right? And then you submit them to Sky Pilot saying that hey, I need eight B300s for example to run this job. And then Sky pilot takes care of the orchestration, right? it looks at your compute, your SEM clusters, Kubernetes clusters, even cloud VMs and then takes care of like orchestrating those resources, running the jobs, getting you the logs, results, etc.

51:30 And you know, Skypallet started a few buildings away from here in the soda hall here at Berkeley. open source pretty active and healthy community of you know, people using it and contributing to it. Also used by some of the biggest companies out there right now. and also it's kind of rooted in the research we did back in the sky computing lab. So just a really quick walkthrough of like how what exactly is Sky pilot, how does it work? you know it's super simple. the simplest way to get some resources in Sky pilot is just to say hey Sky launch give provide what infra you want to run it on and then state what GPUs you need. If I need to switch over from say Kubernetes to AWS, it's just like changing one flag.

52:15 similarly for GPUs again your application needs to be aware of what's running under the hood. but you can easily switch what GPU type is provisioned for you by just changing one flag. and then if you want to do more complex stuff right like not just provision some GPUs but actually have u a complex job like a reinforcement learning run you can we we have this abstraction called job groups where it's a declarative spec. Say you say that hey I have a rollout server which needs these resources needs to run these commands. U then I have to spin up say a thousand sandboxes to run this stuff.

52:52 I also need to for example have the rollout server running on H100s compared to the trainer which may be running on B200s and then it just passes over to Skypilot and then Sky pilot takes care of like the orchestration of this entire workload for you and in doing so it also provides some nicities like for example it'll provide you service discovery life cycle management you know if one of the jobs fails it doesn't mean the entire run needs to fail it'll just like restart that job for Cool. so we've been talking about training, but serving is equally important and serving also benefits from having a unified platform like this. specifically you gain capacity across all your clusters. So instead of having to launch say one case serve instance in one cluster, you can have Skypilot running inference across all your clusters. And when a failure happens, right, like Skyfire can seamlessly move over your replicas to a region where they where you still have capacity.

53:55 And one of the other benefits of sort of like having all your workloads run through one layer is that you can actually do very interesting stuff by running both inference and training together on the same compute. Right? So the idea is you use the same GPU, same giant cluster to run your jobs. But for example, if the inference requests start spiking up, you Skypilot can dynamically reallocate your resources from training to inference. So it preempts the training jobs your checkpointing logic takes care. You don't lose data.

54:25 And then when the inference spike dies out, it will automatically scale down inference again and then continue training. Cool. And I just want to reemphasize all of this is BYOC, right? So it's not like Sky Pilot gets you or like you don't give your credit card to Skypilot to sort of like buy compute. You bring your own compute from your cloud provider, your own data. Everything runs on your own premises. So you know I I'll end here but like really quickly the idea is we we don't need to reinvent or like we need to we don't need to replace Kubernetes or Slum. They're really good at what they do. What we really need is a unified layer which sort of like allows you to bridge the gap between your workloads and all those clusters that you're running. cool. With that, I'll stop here. Thank you so much.