CMU AI Agents 2026: 7. Computer Use Agents

0:00 Okay, great. So, yeah, today I'm super excited to have JY Code as our guest lecturer. So, JY is an expert on on multimodal machine learning in particular computer use agents. so, JY has experience both in academia here as a PhD student CMU working with me and Russ Solakutnav but also has a lot of industry experience. so he worked at Google research before starting the PhD and also he led the computer use agent team at Meta's TBD lab for a year and a half.

0:40 >> Yeah. And so kind of like laid the foundation for the strong agents that they're releasing now. And yeah, he also built some of the most impactful benchmarks in this area and also did work on search for agents. So he'll also be giving a lecture later on in the course and we've covered some of his work already on multi-agent computer use and looking forward to you know hearing about the foundations of KUAS today. Thanks.

1:12 >> Cool. thanks a lot Daniel for the kind introduction. so today I'm very happy to talk about something that I've personally been very excited about for for a couple years now. I think today it's like taken for granted that we have that computer use is a is a core capability in like many frontier models. But as we will see by kind of stepping a bit through the history of computer use agents, this was not always the case. I think for a long time it was not very clear whether we would want computer use agents or how we would build strong computer use agents.

1:47 so I think computer use agents or kas are a little bit different from the agents that you've already covered in this class. so I think they're a bit different I think of them as a little bit different from like sui agents and more textbased agents. so first of all what is a cooler? a coupa is a model is an agent that observes a sorry, it's actually not. okay, there we go. Here you >> I'm I'm think I think it's fine.

2:19 Yeah, sorry. So, AUA is a computer use agent. it's a little bit different from the agents that you've already seen cuz what they take as input is actually the graphical user interface that you have in most applications. so you might imagine this could be a web browsing agent. It could be an agent that operates on your desktop computer or it could be something that operates on a mobile device such as your phone. so all of these are broadly speaking computer use agents. agents that operate through the same graphical user interface that humans use. so they take in like screenshots as inputs and they execute actions of the same type that humans do. So clicking actions, scrolling actions, swiping, tapping and and so on. so the key difference here is that they operate in the exact same space as humans which I think is a very it allows for very interesting problems and challenges to be solved in the modeling and evaluation space.

3:12 So broadly speaking kers operate in the following fashion. You have a user task that is provided to your cooler. You also have some environment observations and typically this is a screenshot of whatever you have on your current computer screen or mobile device. this can also be some other like textbased representations but broadly speaking most frontier course use like screenshot representations. So this is essentially exactly what the user would see if they were to attempt the same task.

3:41 Both of these inputs go into your cool which then produces a reasoning trace typically. So it's like a chain of thought and it produces an action that it wants to execute next. And this action can be any any action that you allow in your in your environment. So it can be clicking, scrolling, typing and so on. We then process this action and we execute it in in some infrastructure that you have in your environment. so if it's a click action, you actually execute it on the computer and you get some new observation after executing this action.

4:15 To give you a concrete example, you might have a task like buy me a bloom map and you might have some initial environment observation for example from your web browser. So these would go into input into your computer use agent model and your model might produce some reasoning trace of the form. I see a blue monk on the center of the screen. I will click it and it will then produce some like formatted action to be executed. So in this case it wants to do a left click on the on the environment.

4:43 we will execute this action on the environment. So we will actually like click somewhere on the environment and we will get a new observation that happens after we execute this action. So for example over here you get a new screenshot. in this case the agent has clicked into the blue mark. So you kind of have a new observation that shows the details of the blue mark. Your model then takes in this new observation and it produces a new action to be executed condition on this updated observation. So it will it might then say like delivery details are below. I will scroll down to find them and it will produce this scroll action that we will again like execute on the infrastructure. you then get a new observation and you basically repeat this until your model decides it's done. Right? So this can go on like for many many iterations depending on how complex your task is.

5:31 so yeah you start with an observation your model executes an action you get a new observation and this repeats until the end and typically you get some reward either from the environment a human user or some other judge telling you whether the model did it successfully or not. so typically like this reward is just a 01 binary reward but it can also be a partial reward in some cases. so this is broadly speaking how a computer use agent works. It's this like observe, reason, act loop that that repeats until your model exhausts its budget or it thinks it has successfully completed the task. and today it seems like very simple and models are actually very capable as as we'll see in a couple slides. but this was not always the case. So maybe we can have a quick poll. So when do people think like computer use agents were first like invented? do people think it was like before 2020?

6:30 Okay, a few hands. say like 2020 to 2024 period is when we started building coolers. Cool. And who thinks it was like after 2024? so the the answer is actually that the first cooler paper came out in in 2017. So this was like quite ancient in in AI standards, right? it was actually a paper from OpenAI. So they had this project called World of Bits where they built this very simplified toy environment. as you can see here, it's mostly like synthetic data. It's very simplified environments that look maybe a bit like what you might see on a mobile device, but it's very simplified.

7:08 not not realistic at all. and what they did in this paper is that they actually did like a bunch of RL training. so this is not really similar to like modern LLM RL but it's more like similar to RL that we did for game agents in in the past. but this was I think this project was like ahead of its time. All of the models back then could not do this at all. Like even this simplified mini world of bits task they were very bad at it.

7:37 so this was the first cooler paper probably. in 2022 I think a bit more interest started picking up with the advent of language models. there was this other paper called web shop that also did kind of like simplified shopping environments but it was a lot more realistic than than mini world of bits as you can see in this visualization here. So it's kind of clicking through this like simulated Amazon site. and it's still a bit artificial but it's a lot better than mini world of bits. and at this point like language model agents started doing a bit better at these tasks. and we weren't really training them in in in the way that we were for many world bits in 2022. In 2024 actually there was this I think of it as a seminal work from actually Graham's lab at CMU called web arena. So it was this work that created a lot of simulated environments.

8:29 and as you can see like all of these environments are a lot more realistic than web shop and mini world of bits. they contain like very realistic data often scripted from real world websites. So these environments were populated with actual data from like Amazon and Reddit and and so on. so these are like very realistic data and environments. and I think they marked a point where the models also started getting a bit better at these problems.

8:54 In 2024 there was also this almost concurrent release of this benchmark called OS world. So I think this is one of the most standard computer use benchmarks these days. And this was also using real apps and kind of like real data but it tackled like desktop use rather than web browsing. So it's a bit more general. You can use a lot of different apps on Linux, Windows and so on. so this is a bit different from web arena because web arena was web browsing but OS world is like fullon desktop use.

9:29 I think things also took a bit of a change like in 2025 to 2026. So this was the point where like coding agents got a lot better and at this point like a lot of people started generating environments end to end. so for web arena and OS world what they did was they actually took open source implementations of these apps and they filled it with realistic data but all of the all all of the software was already written by humans and open sourced in some manner. my PC bench and a few other benchmarks around the 2025 to 2026 period actually use coding agents to generate the entire app and website end to end. So shown here is actually a Gmail clone. It's not the real Gmail, but it looks very close to it. And this is actually generated completely by an LLM a coding agent.

10:15 all of the data that is seated here is also completely synthetic. So, we've actually gotten to a point where a lot of like frontier cooler environments are completely generated and completely filled with synthetic data and I think they actually work very well. sometimes they're even better than real applications because they're a lot cleaner and they're not as as noisy. Ka world was also another project that came from CMU that took like a real world software. So, I think Kool World is very interesting because the software used here is a bit more of the long tail. So you have some software used for like astronomy and biology and other things of that sort. so it's a bit it it departs a bit from like just consumerbased apps into very specialized desktop software. all of the data here is also I think is realistic data. So it's downloaded data from real world use cases and is loaded onto onto onto these environments.

11:10 shown here is a demo very recently from this month of GPT6 Astra. So GBT6 Astra was a huge leap in computer use. I think it basically like solved most of the computer use benchmarks. it's like better than human baselines at this point. it's kind of a lot faster than previous models, but it's still very slow and very expensive. So this demo is showing GPT6 Astra operating like a CAD software. So it's doing like some like 3D modeling. it's actually sped up 30 times. So in in practice like this model is actually 30 times slower than whatever you see here. but but the point is it's very capable but it's still very expensive and and slow. so we still have some research problems left but in terms of capabilities I think current models are are very good.

11:54 so you can see like we progress from toy environments with miniw walk to today 9 years later where we have GPT6 extra that is comparable or better than your median human at a lot of computer tasks. so very exciting very fast progress and we can think about like how we might measure these problems. if you think about them from a research or academic standpoint benchmarks are one of the core ways in which we measure how fast these agents have been progressing.

12:22 so we'll go through an overview of like several computer use benchmarks and how they measure different axes of agent performance. so if we think about how computer use might be measured it's actually a bit I think it's a bit more interesting than some of the other agents that we've already covered in this course. so computer use is in some sense a bit less of a verifiable task compared to like software engineering or math. in software engineering or math you kind of have unit tests or or symbolic verifiers that tell you whether you successfully accomplish a task. for computer use is actually a bit different because you can imagine that successfully accomplishing a problem is not always people do not even people do not generally agree on it. So it can be challenging to write like programmatic verifiers. so we have several ways of evaluating how computer use agents do on some of these benchmarks.

13:16 the most obvious one is whether the action was correct. So you can measure whether a click action was done on the correct item. you can also measure the outcome which is probably the most endto-end fashion. So you give it a task like buy me a blue mark and you measure the outcome of whether it actually managed to purchase the item or not. So this can be done through either like checking the state if you have a simulated environment or you can just look at the trajectory of the agent and decide whether it provided enough evidence that it did the task.

13:49 and lastly, I think you also want to measure whether the model respected any constraints that might be inherent or explicit in the task. So, for example, you might want it to navigate to the checkout page but not actually check out. so you also have some other verifiers to make sure that it did the task while respecting any userdefined constraints. So, I think evaluation is generally very challenging for Kas. there's a few broad categories of evals that we'll go through today. so broadly speaking, I've categorized them into four different categories. So the first one that we'll talk about is static evaluations.

14:23 so let's go back to our task. Add a blue mark under $20 to the cart and do not purchase. Static evaluations measure whether every action along the the trajectory of the agent is correct or not. So for example, you might start off on this screenshot and your agent would, you know, try to attempt to click on this blue mark here. we might have some ground truth reference rule like we want XY that the agent predicts to be in a target box. If it's if the prediction is within a target box for example this red box here we give it a score of one.

14:54 otherwise we kind of give it a score of zero meaning that it did not perform the correct next action at this point. so in this case like for static evals we consider it to be correct if the current step produces the correct action or not. so this this because it's static you can't really like roll out the agent end to end. we just consider actions at every point to determine whether it successfully accomplished the current action.

15:21 One canonical example for benchmarks in this area is known as screen spot pro. so screen spot pro is what we call a grounding benchmark. it tests the abilities of the models to ground actions in the screenshots and it consists of a variety of applications operating systems and about 1,500 different screenshot examples. The evaluation in this case is the perpoint accuracy. So for example this task of constructing a UV spear mesh you have this like Blender screenshot here. So this is a screenshot. It's not an actual environment. So the agent does not have access to any interactive environment.

15:58 is just given this screenshot and the goal is to determine whether it produces an action that falls within this red bounding box over here. So if it if it predicts a click action that falls within this bounding box, we consider it to be a correct action. Otherwise, we consider it to be an incorrect action and and we don't give it any any reward. Another example of a static benchmark is mine to web which tests a bit more of a variety of actions. So you can think of this as like next action imitation in this case given a current screenshot and a task. for example rent a car in Brooklyn central I'm supposed to predict the next action that my agent will take. so in this case the ground truth action is to click on the car tab and no actual action is executed on this task. You're given a series of screenshots and your agent has to predict the next action condition on that screenshot. So once it predicts the action we score it and we provide it the next screenshot from our offline data set. So again, this is not actually executing any actions in the environment. There's no environment in this case. It's a series of static screenshots and ground truth actions.

17:02 so as you can imagine, this kind of evaluation is very fast. It's very cheap to run because you don't actually have to support an environment or simulation. You just have to provide it screenshots and measure whether the resulting predicted actions are within the ground truth set. but one problem as you can imagine with static evals is that there's a lot of false negatives in the sense that for this screenshot and for this task there's a bunch of different actions that you can click that you can perform that would be considered correct like for example I don't really have to click the car tab in this case I could adjust the dates to begin with and that would also be considered a valid action but in static evals because you have a set of ground troop actions we we actually would mark those as as incorrect. so that's one of the main drawbacks of static evals. They are very fast but they don't really measure the thing that you care about which is the true end toend performance. So I think what we actually care about is whether the agent successfully did the thing at the end of the day. which brings us to our second type of evaluations which are end to end evaluations.

18:09 so again like let's let's look at this example task like add a blue monk under under $20. we might have an agent execute this task in an environment or on the real web and at the end we can check whether the agent did it correctly by using some programmatic unit test like checks. So for example an agent might have attempted attempted this task and at the end of the trajectory we can run these like unit test like checks to see whether it actually did it. So this is done in a simulated environment. So we actually have like API access to all of the back end and state of the environment. so we can check whether the blue mark is in the cart whether the blue mark is less than $20 and whether the agent has not actually place the order. So we check whether the order is the order count is zero and we only give it a reward of one if all of these individual unit tests pass. so this this is actually like a programmatic check. You actually have an environment in this case. So the agent is doing like whatever it needs to do.

19:08 so we don't we don't check for the individual actions that the agent executes at every step but we do a check for whether it actually accomplish the task at the end based on these unit tests. So if your unit tests are comprehensive enough you could imagine that this is a very good reward signal because we don't really supervise every single action of the of the model. we just kind of like check whether it did it correctly or not at the end.

19:34 So web arena was one of the first benchmarks to do this. web arena was a simulated web navigation benchmark consisting of a few environments. so you have 800 about 800 tasks on four different apps that web arena hosts and the agent will execute all of these 800 tasks and at the end you kind of have a pro programmatic verifier as we discussed to check whether the agent did it correctly or not. So this is an example task. the goal is to create an empty repository called awesome LLM reading. So the agent will go onto this simulated gitlab repo. So importantly this is a simulated environment. It's something that you host locally. you do not go on the real web at any point in time for for web arena. the agent has to try and accomplish this goal within the simulated environment and at the end of the task we run this programmatic verifier which basically checks for a bunch of conditions. so you you essentially check whether it managed to create this repository or not. and all of these verifiers are handwritten by students by grad students at CMU actually. so they they hand wrote these verifiers for every single task in web arena and at the end of the assuming you know the students did it correctly at the end of the execution you would have a verifier that tells you whether the agent successfully did it or not.

20:57 there there's still a few like false negatives and false positives because the verifiers are not comprehensive enough but on the whole it's a very I think it's a very nice benchmark. It definitely like led the way to people taking and evaluating coolers a bit more seriously. so I think it's important to contrast this with previous static evals because this actually measures the end to end performance. So you kind of just check for the outcome. you don't care about the individual steps as long as the agent did the task correctly. but you could also imagine that these this leads to some reward hacking. So the agent might you know try to accomplish the verifiers rather than actually doing the thing which is a common problem across most cool and and agent evals and training generally. but I think this was a huge step up step up in in agent evals. so they made sure that you did it in simulated sandboxes and they measured like the outcome rather than the individual steps.

21:55 visual web arena was a an extension of web arena that kind of like treated it as a multimodal problem instead. so at this point when web arena was released most models were handling the cooler problem or web browsing problem using accessibility trees which are kind of a simplified version of HTML and visual web arena actually tried to convince people that you kind of want to operate using screenshot based approaches. So visual web arena is an extension of web arena that consists of 910 visually grounded tasks. So all of the problems in visual web arena have to be solved using multimodal representations. and the evaluation is a bit similar. So you kind of have like programmatic checks as well. but you also have some multimodal based programmatic checks. So there's some problems that you can't solve if you rely purely on HTML.

22:45 Another form of evaluation for these benchmarks is instead of using programmatic verifiers, you kind of use an a different LLM to judge whether the model did the task correctly. So given a trajectory of a model, you prompt a different VLM or LLM. so a textbased LLM or multimodal vision language model you prompt it with the trajectory of the model and you ask it whether it completed the task successfully and this is also a very popular evaluation metric because it's not always possible to write u verifiers for every single task that you care about. so in this case you again have the trajectory that the agent completed and instead of a programmatic verifier you have a bunch of rubrics or or maybe you just even provide the task to the to the LLM. so you provide the trajectory of the agent and you provide the task and the the LLM kind of tells you whether it thinks the agent completed the task successfully or not.

23:39 so typically like these judges they produce a short chain of thought followed by a judgment for example pass or fail or sometimes you assign like floating point scores but typically the judge tells you whether you pass or fail or not. so this is a bit more general because you don't need to write verifiers for every single task in your benchmark. but it's also a bit more prone to like false false positives and negatives because you can imagine that your LLM judge can be easily fooled into thinking that a task was successfully accomplished when it was not. but in general this is a lot more flexible and several benchmarks that use this form of judging include web voyager which was one of the first benchmarks that tested the abilities of coolers to navigate on the real web. so this is very different from web arena because what they did is they actually came up with 15 different websites and 600 tasks on these websites and they told the the benchmark is to actually go on the real web on these websites to accomplish certain goals. so actually if you go on some of these websites in the benchmark like there's this recipe site called all recipes. If you look at some of the task that the agent is supposed to do you can actually see some traces on the real web of previous agents attempting this task. which is actually it's not great but you can see some evidence of why we might not want to evaluate agents on the real web.

24:58 but nevertheless like web voyager was still quite valuable because it tested the capabilities of these agents on non simulated environments. So it's basically as close to a real world evaluation as you can get. it gets agents to go and do these things on the real web. for example like find the most start GitHub project for climate change data visualization. they they deploy like who was to go on the real GitHub and do a bunch of search and find the actual most start GitHub repo for this task. and in this case because it's going on the real web you can't really have programmatic verifiers because you can't really query the real web for its state. so the eval here is the VLM judge that we talked about. So the judge which is a different LLM gets the series of actions and screenshots that the agent has has taken and observed. it gets the original task. So it sees this task. It sees what the agent has done and it kind of predicts whether the agent succeeded or not. So we we evaluate this problem with like VLM judges. no programmatic verifiers. so this is kind of like you can think of it as quite wild wild west evals. you deploy something on the real web, you run a judge and you you know at the end of the day say whether it's succeeded or not. But I think a lot of people like this because it's in some ways as close to a real world eval as you can get.

26:22 Online mind to web was a was a different eval that followed up on web voyager. so it's a bit more comprehensive. The judge is a bit more stronger. So they show that the judge actually has about 85% agreement with humans. so it's very close to being human level agreement for for judges. but essentially online Mind2 web also looks at real world evals. so they expanded to about 136 websites and 300 tasks and they get LLMs to go on these 136 real websites to accomplish certain tasks. and they they show that it actually using this judge is a bit more accurate compared to Web Voyagers judge. so they they have a bit more of a complicated judge with different rubrics and and criteria. but essentially this also goes on the real web and it inherits it inherits a lot of problems that going on the real web poses similar to web voyager. but again it's a very nice eval because it's in some ways as close to a real world eval as you can get. so this is completely automated but you have an LLM judging the task. You have an LLM attempting the task on the real world.

27:31 and because the judge is very close to human eval kind of have this automated pipeline of an LLM going on the real world doing something and you have a human-like judge determining whether it succeeded or not. So I think it's a very nice benchmark very automated but again I think people don't really use this recently anymore because it's essentially solved by modern models and I think as cool words have gotten more popular people have also become a bit more careful about deploying them on the real world because these website owners really don't like having like tons of agents ping ping their websites to to do a benchmark yeah you can imagine that if your website got included in this benchmark and you started getting a lot more traffic, you probably also wouldn't be very pleased about it.

28:20 Yeah. so one I think one important thing is that all of these evaluation metrics that we talked about are very they are programmatic verifiers or they are LLM judges. so they are kind of like proxies to determine whether we successfully accomplish a task or not. but human review meaning getting humans to judge whether the agent did something or not is probably still the gold standard. So these papers also ran like a human evaluation baseline where they got humans to review the trajectories of agents and you get a few humans per trajectory to determine whether the agent succeeded or not.

28:57 and at the end you kind of like get humans to vote whether the agent succeeded or not. this is in some ways like still the gold standard of determining whether an agent successfully accomplished a task, but this is very expensive to run. So, it's not really possible to do this for every single task in your benchmark or every single like debug or validation run that you want to do. but this is something that most benchmark creators still do. get humans to attempt the task or judge the task and and determine whether the agent is doing the right thing. but this is very expensive.

29:31 So it's typically just done once or twice for ensuring that your agents or your benchmark is reasonable. So the other type of evaluation that people typically do is they're also end to end evaluations but they're done on like desktop computers or mobile devices instead of of the web. so the web is quite interesting because you can do a lot of things on the web. I think personally most of my time is spent on Chrome as well. but there's a different set of tasks where require you to operate an actual virtual machine or an actual desktop and there's also several benchmarks that measure this in an end toend fashion.

30:13 we've already talked about OS world which is one of the canonical cua benchmarks. so this is a real Linux virtual machine that you can spin up. It consists of about 369 tasks across nine different applications like Libra Office, Chrome and and so on. it's 369 tasks across nine different applications. And for each of these tasks, they also have programmatic verifiers. So your agent has to go into these virtual machines, attempt the task across multiple different apps, and at the end, we run some unit test like checks to see if it succeeded or not.

30:51 So some examples of task in OS world are for example you might want to update this Libra office calcul sheet using receipts in a different folder. So in this case your agent has to navigate to the folder find the receipts it has to enter like the appropriate values in this workbook and it has to save the workbook. So it has to do this navigate across different apps handle like entering items in in this Libra office and basically like accomplish this task based off just an initial user instruction.

31:26 After the agent thinks it's done, we run all of these like unit tests that we have. So we check whether the the spreadsheet consists of the items that we think should be there and we assign it again a 01 score of whether we think it succeeded or not. So kind of like very similar in spirit to web arena but in this case it's using real world apps. So these are apps that people actually use on a day-to-day basis and it's operating again in a full virtual machine. So it's not just confined to to Chrome or to the web anymore. It's operating over this virtual machine that consists of nine different apps. so it's a lot more broad and you could imagine that with desktop agents we can automate a wider variety of tasks than if we just use web browsing agents.

32:11 Windows agent arena was a different benchmark that is kind of like inspired by OS World. but it's a benchmark created by Microsoft to test the performance of agents on Windows machines. so a lot of a lot of like productive work in the world actually happens on Windows machines and I think this benchmark was quite nice because it measured the ability of agents to perform some of these tasks that are very common in enterprise workflows or in other companies.

32:38 Yeah. So it's again like you spin up a Windows virtual machine, you get an agent to attempt a task and at the end you have some verifiers to determine whether it did the task correctly or not. And again like because we're using programmatic verifiers some of these are also very prone to having false positives or false negatives but you know they have their merits as LLM judges do but these these are essentially some of the core end to end eval benchmarks that that people have been exploring from the 2024 to 2025 period. Yeah. most of these benchmarks are actually quite well solved by modern coolers. So for example the recent GPT6 release I think achieves like much higher than human performance on most of these benchmarks. So the community has actually recently put a bit less attention on these benchmarks that I just mentioned and they've started to look at a bit more like longer horizon or specialized benchmarks I would say. So long horizon computer use is this other it it's actually the same task but the problems that we care about span much longer horizons. So some of these tasks can take actual humans hours or even like two to three hours or even longer to complete. so now we're really like stress testing the abilities of kas to complete things that humans don't actually want to do or we're getting them to complete like longer and longer tasks. A lot of the ones that we've seen in OS world the web arena typically take a human about 10 minutes or 20 minutes or less to complete. but some of the ones that we'll see in the following slides can be several hours operating over potentially specialized software. So, we're really getting to the point where we're trying to get them to be better than like expert human use of the computer.

34:29 a lot of these long horizon benchmarks, they follow a very similar evaluation strategy. So, you kind of have this action trace which represents what actions the LLM has taken on this environment. and you have a few different checks because it's long horizon. people tend to impose like more rubric based approaches. So the the version where we have a single outcome reward of whether it successfully did it or not is not so useful for long horizon because it's very sparse. the rewards are very sparse. the agent can be doing a lot of useful work in the interim but fail to achieve a perfect score. so we kind of have like a lot more fine grain rewards for long horizon computer use. there's a few different checks here. So state check is the interim rewards that I mentioned.

35:14 these are kind of like rubric based rewards where they check whether the agent did an intermediate task successfully and you kind of have a process reward where this essentially checks whether the agent completed the entire task successfully end to end. so this is kind of like the ultimate reward but we have some intermediate rewards to ensure that we have some non-zero signal for long horizon computer use. and of course you can always have a final verdict for whether the agent successfully did it or not. but again because we're looking at long horizon computer use this is very sparse. So often times like a lot of models get a zero on the final reward which is why you kind of want to have intermediate rubrics or process rewards.

36:00 Yeah. And and importantly as I mentioned like having a a good deliverable at the end is not always a great evaluation metric for long horizon computer use because it's so long. so you kind of want to have some interim rewards to ensure that the model was doing productive things in the meantime. Odyssey's was a benchmark that we built here at CMU that measures long horizon web browsing. So Odyssey's consists of 200 tasks. we have rubrics for every task. So, we have a set of things that we believe the model should complete. so, for example, for this task, plan a trip to a Palm Springs wedding. there's eight rubrics that the model needs to satisfy in order to perfectly accomplish the task. for example, you need to find a a non-stop flight from Pittsburgh to to Los Angeles. you need to find a drive time within the 9:00 a.m.

36:53 to 4 p.m. window. and you have a bunch of these criterias that you need to satisfy in order to accomplish the task successfully. so Odyssey's kind of like grades performance across many different rubrics allowing you to have partial rewards instead of just a single binary pass or fail outcome at the end. Yeah. So this example I already mentioned we kind of have like eight rubric items and the model needs to satisfy like each of the rubric items in order to achieve a perfect score idea.

37:23 so this is kind of an in between of like it it's still LLM verified for the most part, but you kind of have like fine grain LLM verification because you don't just produce a single pass or fail reward at the end. OS World also came up with a follow-up called OS World 2 and in this benchmark they actually tried to tackle like hourlong workflows. So OS World 1.0 know was a lot of tasks that require just like minutes of of human time to complete and OS World 2 actually has about 108 tasks that that on on median require a human 1.6 hours to complete.

37:59 so this is very the some of these workflows are very complex. they require like doing CAD modeling or 3D modeling or photo editing. so we're getting into the realm of like more specialized software use. on average most Frontier agents that they benchmark required about 300 calls to make. so this also typically it's it's a few hours of like just benchmarking an agent. so I think this gets to a point where it's a lot more interesting because you can see these agents even like productively working for hours at a time. so it's very expensive to evaluate, but it's quite magical to like start in Eval, come back an hour later and you see that the agent is still productively making progress on some of these tasks.

38:45 This is an example task from OS World 2 and I don't know if I don't know about the median human, but I definitely would not be able to do this easily. So I think we're getting to the point where you have like expert level task. so the task here is to build a a cat support bracket and export a model that matches this drawing. So you're provided with this reference drawing on the left and the goal of the model is to be able to produce this thing on the right by operating some like computer aided design software. so I think this is this is actually very complex. it requires like navigating 3D space. It requires knowing like what tools you have available in this environment and you need to be able to sustain work over like probably several hours because a human would also take on the order of an hour or at least like half an hour to complete this task.

39:33 so it has a lot of very interesting things. this is a visualization of a GPT6 a GPT6 model trying to attempt one of the OS world two tasks. So again like this is sped up 30 times. So it's actually much slower in practice, but you can actually see like navigating between a terminal environment, going back to the CAT software and actually like doing 3D modeling autonomously for for about I think an hour of time. so it's actually able to design like these 3D parts quite well. and this is a a very huge leap from the models that we had even like two years ago.

40:08 so we're getting to the point where these models are are very good at attempting many of these like long horizon paths. Yeah. so I'll mention a few other benchmarks. So cool world long is another benchmark from CMU that tried to tackle like long horizon computer use. so again this is very specialized environments across a variety of tasks for example healthcare task, engineering tasks or like just architecture and design task. consists of 200 of these types of tasks and most of them require like 500 or more LLM calls. so again this is a few hours of human work time. and when this benchmark was released early this year most models scored a very very poorly on this benchmark but I think recent models including GPD6 actually achieved like close to 90% performance on this benchmark. So, long horizon computer use and like computer use in general is very, very close to being solved or some might say even solved because these models can outperform humans at a lot of these, general software use tasks.

41:17 so hopefully I've convinced you that some of these benchmarks are useful or or most of them were useful at some point in time. and I think I think hopefully I've convinced you that they are very capable and have very interesting capabilities. they are very good nowadays at automating a lot of different types of tasks. so I think the natural next question is like how do we how are these models built? How do we build these models? How do we train them? And how do we introduce these type of capabilities to them?

41:48 so we'll talk a bit about like how how modeling typically works and also how these models are trained. But I think I'll pause for a few seconds for any questions about benchmarks because I know we covered a lot of different things. >> Yeah. So object for something that has a twoar.

42:23 >> Yeah. So the question is whether any of these benchmarks assign partial rubric rewards to intermediate steps. So for example, if the task is to add something highly rated to the card and you did do that but you have some like irrelevant items that you added, do you get penalized for that? I think that's a good question. I think reward design is very tricky in a lot of these benchmarks. So some tasks in web arena do have these kind of like penalization but a lot of the times like people design these benchmarks so that the answer is kind of unambiguous.

42:54 So the problems are a bit artificial in a sense. So for example web arena might say add the top rated item to my cart and if you do that then you kind of get like a perfect score but an agent I I've seen like agents reward hack by adding a ton of things to the cart to try and accomplish the goal. So it depends a lot on the benchmarks. I think typically people don't really penalize it at such a fine grain manner but it's also because at the point where these benchmarks were created the agents were so bad that I think some of the reward scores were a bit more generous to begin with. so I think it's some benchmarks do have these things but it's not I wouldn't say it's like super common.

43:34 Yeah, I think people tend to air on the side of like giving false positives than false negatives. >> Yes, >> this is not exactly like a benchmark question, but do you think there is like benefit in looking at other not? >> Yeah. So the question is whether there's benefits in using other modalities other than just pixelbased representations like screenshots. I think there definitely is. open AAI has said that at least previous models and I assume Astra as well they use like the accessibility representations of the computer when available. So Mac OS has this like accessibility representation that essentially is for like visually impaired users. it's kind of like XML like representations where they determine where they tell you like this area is a description of a certain item like this area is my chrome window and this area is my dock and so on. so the models at least GPD6 do use these representations when available. I think there is a lot of value in it especially for if you want your model to be more robust.

44:45 I'm personally I personally really like the pure like screenshot based representations because it's very I think it's very general. So you train a model that takes in just screenshot representations. You can deploy it on any device without having access to the underlying internals of the of the device. So you can train a model on like Windows and it presumably gen generalizes a bit more easily to like a Linux or Mac computer because they have different like underlying accessibility representations. for example. but yeah, I think models definitely use alternative representations. Yeah. Yes.

45:20 >> Yeah. My question was around the rubrics itself. >> Yeah. >> I'm assuming rubrics are different for each dots. So are they one is that expert generated rubrics or is that like a synthetic rubric that you generate and then in terms of reward are they how what are ways to mitigate that from the rubrics? >> Yeah. So the question is whether rubrics are generated by humans or I guess synthetically generated by LLMs and whether reward hacking happens with rubrics and how you mitigate that.

45:48 so at least for Odysseies most of the rubrics are LLM generated but human verified. so one common way of like generating these rubrics is actually related to the previous question where you give a model like privilege information. So you give it more information than it has at evaluation time. for example like some hints or some HTML representation and you kind of get it to do better than it would in a realistic eval scenario and based on that you can generate like more accurate rubrics because you have like privilege information. so for Odysseies and I think for a lot of industry benchmarks rubrics are typically these days like LLM generated by by human verified. sometimes not even human verified but verified by a different LLM that that has like potentially different information. But yeah, I think I think this is quite reasonable these days to do like synthetic generation, but I personally still feel that a human has to look at everything that you put into a benchmark at some point in order to make sure that that it's legitimate.

46:51 I think reward hacking definitely happens a lot for training these models. yeah, I think we can talk a bit more about that in the training section, but essentially I think reward hacking is a hu huge problem. I would say that rubric based rewards actually mitigate reward hacking a bit better than outcome based rewards. because you kind of have like these different criteria that you need to satisfy. if for example like if you just want to buy like the top rated item then you can reward hack by adding every single thing on the site to your cart for example.

47:21 Whereas you could imagine that rubric based rewards check for this. Yeah. >> Yes. >> Do you have any sense about whether these academic benchmarks are used for evaluation or also do people train on them whether academia or industry? >> Yeah. So the question is whether people train on these benchmarks in academia or industry. well you're not supposed to train on these benchmarks because you're kind of like testing on the thing you're training on. But I think it's kind of like common in industry to generate tasks that are very similar to your benchmarks. so it's very common for people to produce tasks that are like OS world like or web arena like and train on those things instead. so that it's like in distribution but you're not really training on the same test set. So I think that's very common. cool world also actually has an explicit training set. so for some benchmarks they have like train validation and and ideally test splits that people can train on but you're not supposed to train on the the actual eval but it's I think people do find like proxy signals for essentially doing that. Yeah unfortunately cool yeah maybe the last question and we can move on. Yeah >> I was curious what representation What is the action space?

48:46 >> Yeah. >> yeah. So, the question is what the action space is actually for KUS. I I think we'll cover this very quickly. So, maybe we can just do that in a couple slides. Yeah. But essentially, it depends on a model by model basis and it also depends like what environment you want to deploy the model in. yeah. So, how how do we how is a KUA represented? As you might have guessed by now, as is with most models in AI these days, a cool is basically an LLM prompted and accepting inputs in a certain format. so more specifically, it's a it's typically a vision language model because most models take in screenshot representations. So you have a goal that's typically represented in natural language. Like again sorry to use the same example multiple times, but the goal might be like buy me a blue monk that's less than $20. And you might have an initial screenshot that is based on what your environment initially has. So you have a goal and a screenshot.

49:47 this goes into your your policy which is essentially an LLM. and your LLM produces an action. so typically this is represented as like text. So your LLM produces some structured representation in text of what action it wants to produce. So this might be a click action with X and Y coordinates. again we then execute this action. we add it to the to the context that we feed into the LLM at the next step. we also get a new screenshot after we execute this action. So we get screenshot one after executing action one or screenshot zero.

50:19 We again concatenate this to the context that we already have. we feed this again to the LLM and it produces the the next action that we are supposed to execute. We then execute this action again on screenshot one and get a new screenshot and we concatenate this to the context and we we we again feed it to the LLM. So, you repeat this until until your LLM thinks it's done or until, you run out of budget, I guess. Yeah. so you have this loop.

50:45 it's typically called a react loop, reasoning and action. and at every step, you add in you you concatenate the action that you produce the new screenshot to your ever growing context and you feed this back into the LLM. so it's an interle like vision language interle action history. you have natural language in the form of goals, actions. You also have like pixel inputs or potentially other metadata. and this essentially is like concatenated into this long sequence, this long multimodel sequence that you feed into your language model. so this grows and grows. there's different ways that people used to handle like longer and longer context, but essentially, you feed in like the context, to the LLM at every step and it generates the next action at that point in time.

51:32 of course, like we don't actually feed in the actual pixel inputs. typically for most language models, you would tokenize this in some way. So, you would tokenize like your natural language to become text tokens. You would convert your screenshots into like visual tokens or or some like pixelbased representations. And the nice thing about modern language models is that this is typically just like concatenated directly. So, what it looks like to the model is that you have a bunch of embeddings that you feed into it at every single step. the model produces an action that you then tokenize and you concatenate it to the context. so you kind of just repeat this growing the number of tokens that you feed in at every step. and the model takes in that conditions on the context at every point. produces the next action and you kind of repeat this. So it's a very nice and simple representation I think and and it works surprisingly well.

52:27 Yeah. So you typically have like text that you convert into token embeddings. You have screenshots that you convert into visual embeddings. And all of this is just interled and concatenated and passed to your language model. and and these days your language model is a is a transformer. obviously that produces actions for the next turn. Yeah. So the question earlier about how like what action space we usually use. different models have different action spaces. if you squint they like generally look similar. So they typically have a set of actions that they can produce like click actions, scroll actions or whatever other actions you have in your environment, right? so different models produce them in slightly different ways. For example, you might produce them as like pi autogy outputs or you might produce them as just like two calls or XML.

53:17 and you would have to like process this and pass it into something that you can actually execute on your environment. Yeah. there's a question in the back.

53:48 Yeah. So the question is why do we use screenshots rather than HTML? I think HTML is also a valid well HTML is very messy but people have done like simplified versions of HTML as representations. I guess one reason that most so most recent models do not use or they primarily rely on screenshots. They sometimes still use metadata but to generalize it broadly most recent models use screenshots as inputs. And I think the reason for it is that it's a lot simpler. so it's it's a lot simpler to like take a screenshot of your computer rather than trying to process the underlying like accessibility metadata. you you can like get HTML easily on the web but you can't really do it as easily for like desktop computer use right so for that reason I think people like prefer screenshots a lot more because it's a lot more simplified and I think somewhat surprisingly to some the screenshot based representation has achieved like very very strong performance so it seems almost like unnecessary to use HTML or other structured representations so I guess that that's kind of like Why? it's very simple and it and it works somewhat surprisingly well.

55:03 >> Yeah, great question. I think this was something that you know people did not really accept as readily even one year ago. but yeah, the multimodal models are very good. So it's surprisingly easy to do this. >> yes, another question. >> So if it to pick a button or something. So the loading screen as well.

55:35 >> Yeah. So there's a bunch of like different design choices you can do with regards to like how you can handle these edge cases. sorry the question is to the question is like whether whether you would take a loading screenshot immediately or what do you do when when you execute an action. So I think computer use is annoying and also interesting because it encounter you encounter a lot of these like age cases and infrastructure problems compared to like math or or coding models. it this can be resolved in a couple of ways. I think the most common way that people do this is that you typically like sleep or wait for a few seconds after executing an action. So you would execute an action and you sleep for like 3 seconds before you take the screenshot and execute the next thing. so if you do see a loading screen at that point in time, models are typically trained to wait again for another specified amount of time. yeah, there's more complex infra infra there's like more complex ways of designing models where you kind of like wait until the screenshot changes, but I think the most simple and common way to deal with it is you just sleep. Yeah.

56:47 Yeah. Thanks a lot for the questions. So this is essentially at a high level like how almost all frontier coolers are trained to the best of my knowledge. sorry this is not how they're trained. This is how they are modeled and we'll soon like look a bit at how they are actually trained to do these quite amazing things. Cool. So with training a model essentially like most computer use agents are based on pre-trained vision language models. So training happens at a few different stages like obviously when you deploy GPD6 as a computer use agent you're not deploying I guess it's not obvious but you're not deploying a specialized version of GPD6 that can only do computer use. So it's kind of a general model that can have computer use as one of its capabilities and I think this is like increasingly the way that people do it. so computer use specific training happens at a few different stages. at the pre-training stage where models are trained with like large scale internet data core specific data is typically injected in a few different formats. So you have kind of like your grounding data which is data that encourages the model to learn to ground information in in screenshots. so this data typically looks at like the form of you have a screenshot and you kind of have like a series of bounding boxes followed by coordinates. So what the model is learning to do here is that it's learning to just like learning grounding abilities. So it's learning like if I want to click on the button I have to produce like these XY coordinates. So it's just learning like the most basic capabilities like learning localization, grounding and so on. you might also have some data in the form of like immediate actions. So this is actually very similar to the static evals that we talked about.

58:39 you might have these screenshots, you might have some like highle task and the goal here is to predict the next action that you will you will produce. so this is again like very noisy data. It might even be like synthetically generated. But the goal here in pre-training is just to introduce some form of like localization and grounding ability which the kua needs to do later on in order to be able to produce like accurate click and other types of actions. Yeah.

59:07 after pre-training we typically have a post-training phase. so nowadays like this is typically mixed in with other model capabilities but to simplify it we can assume that we only care about like cooler for now. and in this post- training phase there's there's two different stages. So the first stage is that you typically do supervised fine-tuning to train the model to mimic a human demonstration of of a computer use task. So you have a bunch of like human demonstrations that you can collect or otherwise like scrape from somewhere. and this typically looks a lot like the trajectories that we already have. So the human is given a screenshot, the human does something and you record that action. So it might be a click action that the human does, you record that using some like specialized software that you have. so you have a bunch of these like human demonstrations that essentially some humans record humans doing cool like tasks.

59:59 nowadays it's also very common to make in synthetic data. So maybe you prompt your agent in a certain way or you prompt a stronger model to attempt the task and you kind of like keep the trajectories that you think are good. so this is like synthetically generated by a different model or the same model. and you also kind of mix this in. so this is typically a bit lower quality and noisier than human data. But you can generate this at scale. So you don't need humans to actually produce this data. So it's actually generally still quite useful to have this like synthetically model generated data in your post- training stage.

60:32 the more interesting part I think is after this post- training supervised fine-tuning stage your model has some semblance of a cooler capability. So you can after after SFT you can typically prompt your model and it will do a decent job at most of these cooler tasks. I think the more interesting stage is nowadays it's is it wasn't very common like one year ago but nowadays it's very common for kas to undergo an a reinforcement learning stage. So what happens here is that you kind of drop your cooler in a simulated environment or real world and you give it a thing to do. it it it will do that thing. So it will like try to buy a blue mark and at the end you will assign a reward to it either from like rubric based rewards, programmatic verifiers or just llm as a judge. You assign a reward to it in some way and you kind of do like reinforcement learning to reinforce the successful trajectories.

61:23 so this is kind of interesting because it's like there is no human demonstration for these tasks. You kind of just let the model loose in a sandbox and it tries a bunch of things learns from its successes and failures and it you you repeat this until until it's able to do this successfully. yeah so I think people are very excited about RL these days. but the other stages are also very important to ensure that the model has a strong foundation for doing RL.

61:54 Yeah. So let's talk a bit more in detail about the various stages. So in the pre-training phase as I mentioned you kind of have these like screenshots and the goal of this pre-training data is just to learn very basic capabilities. So you probably learn like grounding abilities. given this screenshot typically your model is trained to predict the next token for very long description that essentially boils down to like predicting the coordinates of items on the page. So you might have this like car label and the model has to predict the xy coordinates or the xy width and height of this bounding box. so it's kind of like very noisy data but data that you can imagine getting in very large quantities at scale. so for example like this screenshot data it comes from the United website and you can easily generate like all of the bounding box and grounding data by just running a simple like JavaScript script on the website. So you can extract like all the bounding boxes. So you can collect this kind of grounding data very easily at scale which is why it's a great fit for pre-training where you want like a a lot of noisy but large volumes of data.

63:01 The other type of data that people sometimes collect is action prediction. so you can collect for example in the same way from HTML pages you can get like the element name or element meta of certain things on the page and you can provide this as as the input to the cool and has to predict again the xy coordinates of this. So this is a bit more like semantic understanding where you can provide like a HTML element and it has to predict what it wants to do or what it wants to click on. so again this is like very low-level capabilities. It's really just teaching the cooler how to understand screenshots and how to ground various items on the page.

63:43 Yeah. So you basically learn like screenshot understanding. you learn localization. So sometimes you learn like where items are on the page. you can also learn like the relationship of items with each other. So we know for example like the car button is to the right of the hotel button and this might be introduced in your pre-training data in Subway. but this is typically generated at scale and mixed into the early stages of frontier model pre-training just so that our cooler has a bit of an easier time later on in in learning to ground and learning to produce actions.

64:16 after pre-training the next stage is I think people typically call it post- trainining so again like we are ignoring all the other data that goes into post- trainining and focusing on cooler data here this typically comes in two forms so you have like human data where you pay a bunch of humans to annotate trajectories so you might give someone the task like find a birya taco recipe I could make in under three hours and a human here like actually attempts this task by navigating on a web browser so over here like this example is from momo Momo web mix where they collected like 36,000 human trajectories. So they collected 36,000 demonstrations of people doing tasks similar to this like going on the web and doing things and and at every point you get the screenshot observation and you get the thing that the human did. so you end up with a bunch of like screenshot action screenshot action trajectories.

65:08 They also generated a bunch of synthetic trajectories that we'll we'll talk about in a bit. but some open- source examples of like cooler data are Momo web mix and also agent net. So both of both of these papers essentially like collected human demonstrations of these tasks. So it's a very nice way to train your cooler to like imitate human behavior. so condition on screenshot is trained to predict the next action in the sequence. of course, increasingly so these days we can generate some like synthetic data for for this part of the training as well. so Momo Momo web actually did this thing where they gave a model the accessibility tree of the website. So I think this was addressed in some of the previous questions but some of these models do a lot better when you give them metadata. So they have a model that is trained to use like accessibility trees and they provide the accessibility tree which is essentially a simplified HTML representation to the model and they let this model like roll out taking actions in the accessibility tree space and they collect the same set of trajectories. so you kind of have this model that is using like privilege information that it might not have at actual eval and inference time. So it's doing a lot better than it would have if it just had screenshot inputs. so you roll out this model, you collect like 105,000 synthetic traces in this way and you can also not not all traces are retained. So you can kind of like look at the traces and keep just the ones that you think are higher quality based on some metric like using a different LLM to judge it or using a human to filter them out. so traces are collected in this way either with human demonstration or or synthetic data.

66:51 and one analogy I really like with the synthetic data version where you have privilege information is that I actually don't know if it's true, but I heard that Tesla actually deploy deploys their cars during training with LA. So they have these cars that have LA installed and they do like self-driving and obviously if you have like 3D LA information, you do a lot better. and they collect those trajectories and they discard like the LAR information and they train their model to just use like the pixelbased information. So you kind of have a teacher that is using a lot better information than you would have at inference time. and I think for a lot of cooler applications, people have attempted similar ways of injecting like metadata or extra information in order to collect better traces. so I think it's a very nice way of introducing synthetic data. Yeah.

67:38 Question in the back. Especially with certain pixel values. >> Yeah. So the question is whether if you do like pre-training does the coordinate representation generalize?

68:09 So if you train it on a Mac, does it generalize to a tablet or maybe even like computers with other resolutions? Right? and I think typically the way that we collect like coordinate data is that it's normalized. So coordinates aren't usually represented in absolute formats. So they're represented in zero to a thousand. Zero zero zero meaning like top left of the screen and thousand thousand meaning like bottom right of the screen. So they're like completely normalized and in my experience it transfers quite well even when the absolute resolutions are different.

68:42 Yeah. Cool. question >> more efficient way of doing it like perhaps a >> Yeah. So the question is whether there's like more efficient ways if you have >> instead of getting the screenshot through it just stoing like values what certain items are in some database you can just >> there are boxes just like a description of what the box does >> yeah I think that's a That's also a reasonable way to do it. I think models these days are not trained as like monolithic coolers. Like if you try using like GPD6 Astro, it doesn't always just do computer actions.

69:37 It will interleaf like bash calls and it will like write code instead. So I think models like typically try to solve it in the most efficient way possible. so they definitely like do use that kind of data at training and also at inference. yeah. Oh, okay. Okay, cool. okay. I'll cover a bit more quickly then. Yeah, basically we have a bunch of like open- source data that we can train models on these days. So, a lot of different examples and the final stage of training is actually post- training RL. So as I mentioned in this stage you kind of like deploy the model in a simulated environment or if you are feeling you know extra spicy you can also deploy on the real web but people don't really do that nowadays. so you typically like deploy the model it takes actions in a simulated environment and at the end it produces you get a reward from the environment or from some other judge and you update the policy based on the rewards. So broadly speaking like if you get a positive reward you upweight the actions that happen in that trajectory a bit higher and if you get a zero or negative reward you downweight those actions. So your model essentially is trained to maximize the reward in an environment and in this case it's like doing it completely autonomously. So no human demonstrations. the tasks that the model is attempting might come from humans but typically it's like operating in an autonomous fashion.

71:05 Yeah. And hopefully it's obvious why we need simulated environments for RL. A lot of the tasks that we care about, we don't really want to execute them repeatedly on the real world. For example, you can only book an airport ride on Uber so many times before you get banned or you run out of money, right? So you kind of want like simulated environments to do reinforcement learning in. and there's several benchmarks that kind of like tackle the reinforcement learning building paradigm. so I think I won't go through these in detail but you can reference them on the slides. but essentially I like to show this graph which shows that the reinforcement learning part of post training is actually very important for introducing like strong cooler abilities. so you start off with a success rate after SFT of about.5 and if you do like reinforcement learning at the end you climb to about like 6 or or more. so it's a very important part of ka training and anecdotally it also introduces some abilities that is hard to do through human demonstration alone. So it's much easier to train a model to operate over long horizons using reinforcement learning because you typically don't have human demonstrations of very long tasks.

72:18 It's very expensive to collect those. Yeah. so to summarize yeah we covered like how coolers are trained. you typically have a few training stages like pre-training post- training where you do like behavior cloning on human demonstrations and synthetic data and you also have like the final cherry on top where you do the model reinforcement learning phase of running it in simulation and getting rewards in the environment and hopefully by the end of this you get a strong cool. but at a high level this is like the pipeline that most labs use to train cooler models these days.

72:56 So to summarize kas are a bit different from other agents. I think they are kind of like really nice because they operate directly on the same interface as humans. it's very tricky to evaluate coolers and people have been doing it for for a while. but essentially we want to target the right metrics and level of behavior. cooler models are designed so that they predict actions from interled visual history. So you kind of have this interleaf context of screenshots and actions that you previously took and the model has to condition on this to produce the next action.

73:28 models these days are trained in multiple stages. So you have pre-training with large scale noisy data. You have SFT where you try to behavior clone human or synthetic demonstrations and you have a final RL phase where you let the model roll out itself in simulated environments and you reward it for good behaviors and penalize it for bad behaviors. essentially that's probably the summary of all I have time to cover off cover today but I encourage you to visit the slides for some remaining takes on what we can still work on for Kers.

74:02 essentially I think speed is still an issue. Personalization is a very ripe area for research. infrastructure and UX is a big concern. And, I'm also very excited about multi- aent systems. Yeah. thank you for listening and happy to take questions for remaining time or Yeah. God bless.