transcribe

World Models for Robotics and the Compute Crunch | Oliver Cameron, Co-founder & CEO, Odyssey

Core Matter: Physical AI, Robotics, Supply Chain · 40m · transcribed 7d ago
More from Core Matter: Physical AI, Robotics, Supply Chain Business
𝕏 Share ▶ YouTube 📥 PDF 🤖 .md

Section Insights

# 0:00

Introduction to Odyssey-3

What is Odyssey-3 and its significance?

Odyssey-3 is a foundation world model developed by an AI lab focused on understanding the world through visual observations. It aims to learn physics, dynamics, and human behaviors, making it applicable to various fields like robotics and driverless cars.

  • Odyssey-3 represents a significant advancement in world models.
  • It can be applied across multiple applications, enhancing AI capabilities.
  • The model's physics accuracy is crucial for its adoption and value.
# 8:04

Real-World Testing of Odyssey-3

How does Odyssey-3 perform in unpredictable real-world scenarios?

While Odyssey-3 is not yet ready for full deployment, it has shown surprising capabilities in handling unexpected situations, such as animals on the road, by applying learned concepts from pre-training.

  • Odyssey-3 can adapt to real-time challenges using pre-trained knowledge.
  • The model's ability to handle unpredictable scenarios is promising.
  • Further development is needed before full deployment in driverless settings.
# 16:08

Comparative Performance of World Models and Language Models

How does Odyssey-3 compare to existing language models?

Odyssey-3, with 14 billion parameters, outperforms many language models in specific tasks, particularly those related to physical world interactions. However, language models excel in long-term task execution and reasoning, highlighting the need for world models to evolve.

  • Odyssey-3 demonstrates superior performance in physical tasks compared to language models.
  • Language models currently have advantages in reasoning and persistence.
  • Future improvements in world models are necessary to match the capabilities of language models.
# 24:12

The Role of Data Companies in AI Progress

What impact do data companies have on the development of AI models?

Data companies play a crucial role in advancing AI capabilities by providing the necessary data and operational support. Without them, the progress seen in AI models over the past year would likely not have occurred.

  • Data companies are essential for the growth and capabilities of AI models.
  • The logistical and operational challenges they address are significant.
  • The ecosystem of data companies drives more progress than often recognized.
# 32:17

Adapting AI Models for Various Hardware

How is Odyssey-3 designed to be compatible with different chips?

Odyssey-3 is engineered to be flexible and optimized for various chips, allowing it to leverage specific features of each chip. This adaptability is crucial given the scarcity of compute resources and the need for efficiency.

  • Flexibility in model design is key to optimizing performance across different hardware.
  • The ability to port models to various chips reduces barriers to innovation.
  • Future chip developments will need to prioritize ease of adaptation for AI models.

Transcript

0:00 Historically, you'd need a team of 500 people. And it was one researcher, really, that did most of that work. Deals that we had handshake were the next day disappearing. Compute will become more scarce, will become more expensive, more constrained. Is there a type of data that you feel like, oh my gosh, it's so hard to find? For people that are not familiar with Odyssey, what is Odyssey-3? So, Odyssey, we're an AI lab and we're really focused on world models.

0:32 We believe this is a new type of intelligence that can go many layers beneath language. It can learn the world, it can learn physics, dynamics, cause and effect, human behaviors, all from visual observations of the world. It learns to predict how the world's going to evolve over time. And so, we've been focused now for three years on world models, inspired by our time, myself and my co-founder, in driverless cars. And what we just released is Odyssey-3.

0:59 This is a foundation world model. And what that really means is it's a world model that can be applied to many different applications, whether that's robots or humanoids or driverless cars or drones. It's a model that can solve many different applications. And what's really exciting about Odyssey-3 is that it's state-of-the-art on a number of different benchmarks that we really care about. We really believe that the hill to climb in world models is physics accuracy, that if a world model is increasingly physics accurate, it's akin to language models getting better at coding.

1:30 It's sort of the paradigm that will drive adoption and drive value in the space. And so, yeah, we released that model not too long ago. And the reception's been incredible, in particular from physical AI. Yeah, congrats. I played around with it. And it's super fun to see, like, hey, prompting. And then the whole world is generated. And I saw some of the posts on X. I think you did that as well, where there's this fruit fly that's, like, flying.

1:57 So, that one's my favorite. So, what is the biggest breakthrough? What was the most challenging thing to kind of crack? Absolutely. So, the breakthrough really is that in robotics and driverless cars and pretty much everything in physical AI, the paradigm to date has been that you have a task that you want your robot to do. And what you do is you go collect a whole bunch of data of those tasks, right? So, you want your robot to do pick and place.

2:25 You'll go collect a lot of data of pick and place. You want your robot to drive in the world. You'll go collect lots of driving data. And what the model is exposed to at training time is just that narrow view of the world. All a driverless car model learns is how to drive. It doesn't learn pick and place. It doesn't learn everything else that we can do in the world. And so, that's the old paradigm. That's how things have been done for some time.

2:52 And what a world model really promises is to say, hey, actually, the base of knowledge for any robot that can do anything should be very general. It shouldn't be bound to a single task. It should be that if you want to build a pick and place robot, that the intelligence has knowledge of everything that you could possibly imagine in the world, whether that's running, cycling, driving, typing at a keyboard, everything you could possibly capture on a camera of the world of humans interacting with it.

3:25 A pick and place robot is just one example should learn from. And so, that is the breakthrough that a intelligence now, when it's applied to pick and place, has knowledge that's so vast that it can also increase the performance at pick and place. And this is akin to how humans operate in the world, right? So, let's take driving. When we go to learn to drive, we're not going in to the first time we, you know, sit behind the wheel with no knowledge of driving, right?

3:55 We have knowledge of watching our parents drive, watching other people drive. And we also have this understanding that driving is dangerous, that you're operating at high speed. We also have knowledge of, you know, cause and effect that when you pull the steering wheel to the right, the car turns right. We have this understanding of danger, that we can hurt ourselves, we can hurt others. And that means that when we truly, when we get behind the wheel of a car, we can learn to drive in 30 hours or less.

4:24 And so, an artificial intelligence historically has never been capable of doing that, right? You could never just show a model 30 hours of driving and then it is able to drive. And so, what we're seeing in world models is that dream is finally coming true, that you can show a world model that's pre-trained on, again, everything you could imagine, 30 hours of driving instruction, then it could drive. Because it has that general knowledge at the outset.

4:50 Yeah, amazing. And speaking of driving, you took Odyssey-3 and tested it on busy streets in India. That's right. How did that go? Yeah, I mean, it was remarkable. I think the comparison would be to get similar performance to what we saw. Historically, you'd need a team of 500 people. You'd need, I don't know, two years of development, something like that, to produce the same performance that we saw when we just applied a singular general world model to the task of driving.

5:21 And it was one researcher, really, that did most of that work. And so, we were shocked, frankly, by the results, how capable the car or the model was at driving in India. And the reason we chose India is because, really, what you want to be able to test a world model on is not the sort of basic driving scenarios, right, that drive forward, turn left, turn right. You want to show it can handle cows running into the street, people running into the street, the craziness of, you know, downtown traffic in India.

5:52 That would sort of be a sign that the model has learned very general concepts that it can then apply to the task of driving. And so, that's what we saw. And so, yes, we were astounded by that. Second thing we were really surprised by is that the way that we train these models is, again, you take this very general pre-trained base, this general world model, and that's Odyssey-3. And then what we want to do is to say, hey, Odyssey-3, we want you to output something very specific.

6:21 We want you to output steering commands, right, turn the wheel this way, press the pedals this way. And so, we need to show it some examples of that task in action. And so, what we did is collect 30 hours of driving, of real-world driving, of people steering wheels and pressing pedals, and that was how we trained the model to then control a car. What we also exposed the model to in a sort of A-B test was purely 20 hours of simulated driving data.

6:51 So, not just now real data, but just purely simulated driving experience. So, no 30-hour real driving. Correct. Just 20-hour simulation. And this simulator, we should really share some videos of it. It's incredibly basic. It's just kind of like Roblox for simulation, very blocky things driving. And what we saw is that, yes, there was a performance drop-off, as you'd expect, but that performance drop-off was not at all as significant as we thought it would be.

7:22 It was incredibly tiny relative to the data difference that we were seeing. And so, what I think this communicates, really, is that the model in that experience that it's trying to be exposed to, the driving experience, it's not really paying attention to the pixels. It's not being overfit to, like, the pixels of a car driving and then extrapolating just from that 30 hours. It's really saying, I'm learning concepts that exist in that 20, 30 hours of experience I'm exposed to.

7:56 And in simulation, the concepts are just the same as the real world. It's a person steering a wheel, that navigating along a road, stopping for obstacles, those sorts of things. Wow. And so, yeah, I think it promises that at pre-training, a world model is just learning real concepts of how the world works that can then help drive a car or all these other crazy tasks. That's fascinating. On the busy streets in India, did it trip up in any time?

8:24 Like, let's say, an elephant came out or things that shocked the model that they couldn't handle? Yeah, I mean, it definitely was not perfect. You know, we're not claiming that the model is ready for primetime deployments in fully driverless settings. But I think, in fact, the things we were most surprised by were its ability to handle those crazier cases. We released some clips of, again, those cows running into the street and U-turns and things that were happening that were definitely not in that 20, 30 hours of experience.

8:56 And so, again, in that 20, 30 hours, we didn't have any cows there. And so what it's then relying on is what it's learned at pre-training is that cows exist. They walk like this. Their speed is this. And then it could apply that to a real-time setting as the car is driving that, oh, a cow. I should avoid a cow. I should turn this much because a cow can move this fast. Those sorts of concepts. Awesome.

9:21 Seems like it's already driving better than me. So, well, you guys are working with a bunch of robotics companies, and one of them is Flexion. So you guys are developing humanoid control policies with them. And tell me more how that works. Yeah. I think companies like Flexion are a great example because the role we play in physical artificial intelligence is that we accelerate every physical AI company, whether it's a humanoid company, a robot company, a driverless car company, a drone company, new form factors.

9:58 We don't compete with them. We don't deploy robots. What we do is we provide them a pre-trained intelligence that they can then build upon. And that pre-trained intelligence is very general. It's very broad. What they do, companies like Flexion, is say, hey, I want to make humanoids amazing. I have this fleet of humanoids. I want it to do these tasks. And so what they'll do is they'll post-train a model like Odyssey-3 to their task, to their robot.

10:26 And it will take the base model and make the performance much, much better at that task that they want. And then they deploy the model plus the robot into the world. They sell to many companies to deploy their robots. And so, yeah, companies like Flexion are amazing and their technical teams are incredible. So they're able to move really fast in taking these pre-trained intelligences and getting them out into the world. Amazing. I think I'll add about this is I think there's been a change in physical AI that didn't used to happen, which was that, yeah, if you were building a driverless car company 10 years ago, there was no chance that you would not build everything yourself.

11:07 You would feel you had to. There was no other way but build every single thing yourself. And today it's very different. I think language models have shown people that there are going to be companies that build foundation models. And then there's going to be companies that build on top of foundation models. And so we're seeing this now in physical AI where a foundation model company like Odyssey can provide that intelligence that means the people building on top of the foundation models can go really fast.

11:37 And they don't feel like they have to do pre-training themselves or these very expensive, tedious things they can just build on an existing base. That's really exciting. For someone that, like, I'm not actively training robots, it's always kind of difficult to know, like, how does one model compare to another? Yeah. And I know that you guys worked with Poke & Wiggle to evaluate your model and to understand more, hey, like, what are the strengths and weaknesses there?

12:02 And tell me more how that works and how does that help you guys inform the next phase of development? Yeah. I think building on my last answer, the other interesting aspect now is that the physical AI ecosystem is booming. You've got companies growing really fast in foundation models, in applications, in infrastructure, in data. And Poke & Wiggle, I think, is a great example of a infrastructure company.

12:33 They're enabling companies like us to better understand the capabilities of our models across hundreds of different robot form factors. And previously, what we would have had to have done is build a robot farm, right? We would have to spun up a big warehouse, put all these robots in, maintain them, set them up. And that just takes a lot of time, a lot of money. And instead, we pay a company like Poke & Wiggle to go and do that same thing for us.

12:58 And so, yeah, I think those companies are going to see a ton of growth because you just get better understanding of how your models perform at tasks. How do you work, like, tell me more, kind of like the process in terms of how you work with someone like Poke & Wiggle. Do you send them the, like, give them access to model and then they would do their magic and then tell you the test result? So, in software, you have test suites, right?

13:27 You have a test and then you have your software and you can automatically figure out if your software passes those tests that you defined. And it's much the same with a company like Poke & Wiggle. We have our tests, things that we want to understand if the model can do, and then we send them a model. And then those two things come together and we get back after a few hours all the results. How did our model do against all of these different tests, all of these different permutations of the tests?

13:58 Did we improve over last week or did we regress? And so, it's quite straightforward as to how this works. And that masks a lot of the complexity because, again, in software, right, you can run these tests just in a cloud somewhere. These are running, these tests are running on real robots with all the complexities that come from that and real-world tasks as well in those robots. And so, yeah, we're very thankful that companies like Poke & Wiggle exist.

14:27 Were you guys doing that evaluation process in-house before these companies come into existence? Yeah, everyone. And, you know, Poke & Wiggle is not alone. There's lots of companies being started. I think it also correlates with the explosion of robot form factors. It gets very difficult for any one company to understand truly how many robot form factors there are out there, right? There's so many robot developers now. It's amazing. And so, you know, Poke & Wiggle prides itself on saying we have this ability to procure every robot that exists out there, get it set up in this factory line, almost, so that any robot developer can test on it, and that's a huge advantage for them and something we don't have to do now.

15:11 In the past, absolutely, we would have had to have done that. Wow. I'm just imagining, like, all the robots sitting in their office. That's fascinating. And I know that you mentioned language models, where lately with Astra and, you know, all the capabilities that's been coming out on the LLM side, they're pretty powerful when it comes to controlling robots. And how do you see these LLM and world models like Odyssey-3 play together in the future when it comes to deployment, physical AI?

15:45 Absolutely. I think it's a very nuanced topic, and I think maybe it's worth starting with how to make this an apples-to-apples comparison. And so, right now, what we have is a model like Astra, which is incredible. Who knows exactly the right parameter count, but rumor is about 5 trillion parameters. It's very big, right? Very big model, very smart model. And then the state-of-the-art in world models today depends on exactly what metric you're measuring.

16:20 But Odyssey-3, 14 billion parameters. Okay. And so, small world model today, very big language model. And firstly, models like Odyssey-3 beat the language model at many tasks. Even though it's, you know, Astra is dramatically bigger, it still beats with a world model that's natively trained to operate in the physical world, which is very exciting. But secondly, there are tasks that models like Astra can do that typically play to the strengths of the language models today, which is mostly the fact these models can operate over long time horizons.

16:58 They can try stuff out, they can fail, they can keep going, keep going, keep going. And as agents in that language model world get trained to operate longer, that means that they can just keep persisting in the physical world too. And so, that's where today the language models, I feel, stand out. They can, in robotics, just keep going. And if you don't really care how long a task takes, it might find its way to a solution.

17:24 Whereas a world model might have given up long before because we don't yet have this reasoning paradigm that the language models have. And so, to make it an apples to apples comparison, I think what we need to do is two things. One is that we do need, and we're working on this, a reasoning paradigm for world models. This idea that the world model can have this assumption about a possible future. It can try it out. It can fail and it can recover and it can just keep going and going and going over time.

17:53 Second, though, is simply we need to scale world models. There is no reason a world model should be 14 billion parameters. It can be any parameter count you want, and it should start to approach the language model parameter counts. And so, I think the apples to apples comparison will come from a world model that has reasoning that is scaled dramatically beyond where we are today. And when you have that, I think it will become much clearer that world models are it for physical AI.

18:22 They're just trained to better understand the physical world, have a deeper understanding of physics. They don't have this sort of text abstraction that gets lossy when you're physically interacting in the world. But it's worth being paranoid that language models are, of course, increasing in capability very fast. But, yes, our research is squarely on scaling world models and introducing reasoning. Amazing. So, you mentioned that the next stage would be adding reasoning and also scaling the parameters to not just billions, but trillions.

18:58 And how does it look like when it comes to acquiring data to train them? What would you need to get there, right? Absolutely. So, we started the company in late 23, and in trying to learn the world, it felt like there was a data deficit, right? There wasn't an internet equivalent that you could go and scrape, per se. And so, what we had to do was go collect data in the physical world. And so, in early 2024, we sent people out into the world with these backpacks.

19:31 These backpacks had crazy good cameras on them and were effectively an early version of this boom we've now seen in physical AI data, where people all over the world are strapping GoPros to their heads and collecting lots of data of humans in the world, right? And so, we started that process early to learn the world. And thankfully, now there is this huge ecosystem. There is a huge, massive, growing ecosystem of companies that are collecting physical AI data and then selling it to companies like us and to other labs.

20:04 And this is a huge tailwind for our company. The idea that there is just unprecedented scale of task collection across every task you could imagine was not a thing 12 months ago. And so, that's a huge boom for us. We buy from lots of these companies. We trust that they're going to keep growing fast and keep providing us more data. And so, that's a huge contributor. Secondly, though, is that really, you can have all the data you want, but if your model is not able to learn the right things through observations of that data, what's the point, right?

20:36 And so, I think what we've also been heads down with and feel very good about now with Odyssey-3 is that we've understood the right training recipe to really take all of this data and to produce a model that has an intuition about physics that's increasingly accurate. And so, we feel very good about our progress in that training recipe to really produce a world model that is understanding the world, that is learning truly how these facets of the world work.

21:06 And so, that's all to say data is very important. The algorithms to learn from said data, that matters a lot, too. For sure. And do you guys still send people out with backpacks? We don't need to anymore. Yeah. There's just enough companies now doing the same work on our behalf. Sounds like a fun job. It is. I did it early on. It was very fun to hike and to look at the data after and just how amazing the world is.

21:33 And I think, but now, there is this ecosystem. Yeah, amazing. And is egocentric data the main data diet that your models consume? Or do you guys use, like, gaming data or custom footage or internet data? And what type of data do you spend the most on? So, egocentric is very important. Humans operating physically in the world, that's very important. But we do believe that that can also lead to a path of a narrow model.

22:04 If you just learn the perspective of the world from a human perspective, you're going to learn something about the world and how it works, but you're still missing stuff. And so, we also believe you have to learn the world from all observations, all possible observations, whether that is humans and egocentric, whether that's just general video of the world unfolding. There's so much else to learn from that, that we believe that's important, too. And so, really, you can think of it as a data mixture.

22:37 And egocentric does play an important part. But we also believe there's many other components to learn from other types of video as well. Would you say that it's still, like, more than half? It's a little less than that. Okay. I think, at least for us, what we're optimizing for is not any one application, right? We're building a foundation model, and so that really necessitates us truly providing a foundation model. And, again, you can sort of – it would be like us, 50% of our data mixture being driving.

23:07 That would encourage the model to learn driving more than other tasks. And so, we really try hard to be quite scientific about the right mixture of data to encourage the right tasks, things like that. I'd be very curious, actually, in a modern language model, how much of the training data is coding. I bet it's still, even though it's the prime application, not 50% of the pre-trained base. I would posit it's less than that. Yeah. Yeah. Yeah.

23:35 I'm curious to find out, too. Is there a type of data that you feel like, oh, my gosh, it's so hard to find that would be great if that's available easily and you can use that for your training? Not anymore. Okay. Honestly, if you had some crazy suggestion, I bet I could go to any of the 10, 12 companies that we work with and say, can you go get this? Three of them would raise their hand and say, I'll go get this tomorrow.

24:00 Wow. And I saw a video. It's a little bit dystopian. It's not one of our partners, but I saw a video of grandmas in an old person's home wearing GoPros, sewing. And so, I hope they're being paid for this. But they're collecting videos of grandmas sewing. Models are learning from grandmas and how well they sew. And that's making its way to foundation models. And so. It's incredible. Yes. Incredibly.

24:31 Incredible and a little bit dystopian. But, yeah, I think this ecosystem. Put another way, forget world models a second. Just think about the progress in language models. I think if you were to, a year ago, stop the rise of these data companies and just say, there are no more data companies anymore. Your foundation model companies, you're just left to yourselves to figure this out. Would we have seen the progressing capabilities that we have seen in the last year?

25:03 I would say not. I think this ecosystem of data companies has driven more progress than we give them credit for, I think. Yeah, for sure. And it's a lot of logistical work, operational, you're super heavy. Like, everything from hardware to humans, different countries. Absolutely. The RL environment work, lots of handcrafting behind those environments, for sure. Yeah, for sure. You know, as Odyssey-3 and Odyssey 4, 5 continue to get more powerful, one of the things that you mentioned earlier is the comparison between the 30-hour real-world training data and then the 20-hour of simulation data.

25:42 That is more like a Roblox type, very low-fi type. And how do you think about the need, like the demand curve for data as the model gets more powerful? Like, do you ever see a world where, okay, like maybe when the models are more powerful, then you just need to use simulation data from there on? Yeah, I think a big part of our thesis is that world models are the ultimate environment to learn within. And so really, if we were to boil down the two killer applications of world models, one is a world model to act in the physical world, robots and so on.

26:21 Another is world models to train agents. And that maybe a lot of the spend that's currently going towards RL environments, it's going towards task-specific data, should be, in fact, now redistributed to world models. Because maybe placing a language model inside a world model is the ultimate RL environment, is the ultimate data generator for a language model. And so, yeah, our belief is that at some point the world model becomes so capable that any intelligence that wants to improve its performance should just be exposed to a world model to learn within.

26:59 And it's not crazy because, I mean, human intelligence in large part has evolved as a consequence of the world model that we live in, systems that we've set up in the world that shapes our intelligence, shapes our capabilities, our skills. And so why should AIs be any different, you know? Wow, that'll be really interesting. So, like, both on the training, but also, like, collecting data. And so it's kind of this infinite loop of increasing intelligence.

27:29 Absolutely. And, you know, one of the things that definitely have been top of mind for both robotics companies, but also, like, just general, anyone working in AI is compute. Yes. And you mentioned that access to compute became top of mind for you. And when did that point happen? And how do you think about it right now, running a company that definitely needs a lot of compute? It was towards, I would say, March this year, 2026. And this was around the time, might be a month or two out, but around the time that OpenAI and Anthropic were both raising $100 billion.

28:05 I think maybe $80 billion in Anthropic's case. And they took that capital and very quickly went out and did a number of big compute deals on the back of that. And so we went from a world of, I wouldn't call it abundant compute, but some availability of compute to a world of very, very constrained compute. Because, let's say, $100 billion of that $180 billion was put out into the compute ecosystem saying, I want to reserve and buy everything you have as soon as you have it.

28:39 And so the dynamics just changed very quickly. And so it became existential for us, I would say, yeah, around May, June, where what we saw was really deals that we had handshaken, you know, handshaken, were the next day disappearing. Wow. And so we then treated it as a code red. It was something that we needed to really figure out to make sure we do have the compute we need to be successful. And so we did a number of deals that I'm very proud of having done now that were very uncomfortable at the time because the terms are changing every day, the deposit amounts are going up, everything's changing, the compute availability, the locations of the compute, those sorts of things.

29:22 And very thankful we did those deals with Amazon, with AMD to get access to the chips that we need. Wow. I mean, it's crazy to think that, you know, a company, your scale, like, which is already, like, pretty big scale, that still think about this as an existential threat. And did that change or how did that change in terms of how you think about fundraising and, like, both the most recent round but also in the future?

29:48 Definitely changed that it feels like simply wanting to procure compute is no longer enough because what we found is that if you are not a partner of these hyperscalers or neoclouds, the only lever they have to choose who they go with and the hyperscalers are choosing the companies they partner with, right? That's a different dynamic than most, you know, buy and sell systems. They have so much demand that they can actually choose who they work with.

30:18 And so then the question is, well, what other value beyond the money you have are you providing to that hyperscaler in this dynamic? And so we treated our fundraise, our series B, as a sort of partnership search process and to say that compute is a part of that process to choose and determine the chips we use and whatnot. But also, let's also partner closely with these companies. Let's partner closely with Amazon, with AMD, with NVIDIA. And I think that partnership plus the compute commitment was important to us and to them to ensure that we get access to chips not just now but in the future as well.

30:58 Because I think actually there's almost a lot of copium going on where there's this hope that, you know, compute's inflated right now and that it will get better over time. And naturally, humans think that way, right? It's how we can get comfortable with things in the moment. I think there's equally a case to be made that it will actually get dramatically worse, that compute will become more scarce, will become more expensive, more constrained. And so I think, again, we still operate today with this mindset that compute is going to become harder to get our hands on.

31:35 Is that your take as well? Yeah, absolutely. I very much believe this will not be easier to get access to but harder going forward. Wow. Speaking about chips, AWS is your main cloud right now. And you're tuning your models using Amazon's Trainium chips, as well as AMD and NVIDIA, both are investors. So walk me through your thought process in terms of how you choose which chips to use. Yes. So we operate this hybrid fleet, we call it, of chips, which is NVIDIA chips, Amazon Trainium chips, and then AMD chips.

32:13 And we find benefits to all of them. I think what we do is to say, hey, for a particular part of the process of producing these models, there's just certain chips that really shine in those particular tasks. And so we then optimize our models for those particular chips. I think also, though, there is just a realization, going back to the compute scarcity discussion, that the time where you could be rigid about your model or your engineering, purely optimizing for one chip, good luck.

32:50 You know, I just think you're going to find yourself really boxed out of many opportunities if your thing only optimizes for one chip. And so we're very much taking the perspective that we should be building technology that can be trained on any particular chip, that could take advantage of the particular features of that chip, and building something universal like that. Did you have to kind of re-architect the model in some ways in order for it to be portable to various chips, or, you know, how big is that process?

33:23 I mean, there is definitely an ability to tune a set model, kernels and such, to a particular chip and get the best performance from that chip. I'll say language models have really changed the dynamics here, you know, there was always this notion of CUDA lock-in, and that was, I think, a bit overblown, the term lock-in specifically. But the idea that now we can simply have coding agents help us port our models to different chip sets, that means that some of the bottleneck that used to exist in saying we can't possibly think about adopting this exotic new chip type.

34:04 Also, I think the chip companies, the new chip companies in particular, they know that they can't make it difficult to port. They know they have to make it pretty straightforward, and that just reduces, yeah, lots of barriers, too. Yeah, that's interesting to think about the future of chips, like, which chips will be the up-and-coming, like, challenger. One question that I always have for talking to, like, robot makers is that, you know, as models like Odyssey-3 gets more powerful, they can use that to operate their robots.

34:38 So, if that's the case, like, what is the advantage of, like, what is the competitive advantage of a robot company, in your opinion? I think many things. So, firstly, is the training or the post-training of the model. And so, if you're a robot lab and you have, again, a particular task, a particular robot that you want to deploy, you can post-train a model really well or really poorly, right? And so, I think, yes, the best robot labs, the best robot deployers will just build this muscle of post-training.

35:14 That's definitely one. The second is that, I mean, the complexities of real-world deployment, getting robots out into the physical world, maintaining those robots, improving those robots over time. That's a very difficult task for many of these companies. And so, they'll build a strength there, too. And thirdly, probably, is the economics, right? The idea that you can constantly drive down the cost of these robots over time. You can get your process of deployment to become more economically viable.

35:44 I think that's another place they'll likely differentiate. And I saw much of this in driverless cars. You know, there was a lot of celebration when the benchmarks showed that our AI was better than a human. But that was really the first step of a very long, tedious journey to then get the benefits of that into the physical world. It's just a complex place. A full of rules and people and challenges. And so, the robot labs, I think, have to build that muscle to do that.

36:17 The past decade, you've founded multiple companies. What drives you to start another company? Well, I mean, there's something inherently irrational about starting a company, right? I have this very inconvenient trait of being quite forgetful. And so, one of the things that I think is shared amongst founders is that you forget the pain of building things very quickly. And so, that helps, I guess.

36:49 I think, though, right now, what drove me and my co-founder to start Odyssey was that it felt like there was this momentum behind language models in 2023. And rightly so. Incredible models. But, I mean, it would be quite a sorry state of things if that was it, right? That there was no other attempts at something else. And so, it felt like in 2023, the world could go down two paths. One is that everything just looks like a language model.

37:22 That's the only way this could be done. Or two, let's also explore other things. Let's see if there's ways to go beyond them. Let's see if there's ways to complement them. Let's see if there's just some other intelligence out there. And so, I think that curiosity of the world and how it works and how we as humans have learned and operate systems in the world, I think that drove a want to do something a little different than language models.

37:47 How big is the Odyssey team now? We're close to 80. 80 folks, yes. It's been growing very fast. It's really fun. Yeah, and what is one mistake that changed how you lead? Oh, wow. One mistake. So many. I think that it goes back to Voyage and how, again, the bet in particular for artificial intelligence should always be that you solve the hardest thing first.

38:17 I remember it every day because, yeah, that was just an extremely painful mistake to have made. And I think it's very tempting, again, as a human to not solve the hardest thing because it's just in our DNA to progressively solve problems, right? And I feel so strongly now, any founder that's in particular in deep tech, if you're finding yourself tempted to solve this sort of like stepping stone problem, probably means you're doing it wrong.

38:49 And you should just go full on out for the hardest possible permutation of your thing. It's maybe terrible advice. I don't know. But I think even just mentally being willing to take on the hardest thing first and go through all the pain to get to that outcome, which may be a good outcome or a bad outcome, I think that mindset is probably the right one to have. Yeah, it's super inspiring. And also, you know, with the power of all the technology around us, it's even more of a good reason to shoot for the most ambitious form of whatever that we're doing.

39:23 To wrap up, outside of work, like what makes a good day for you? A great day. I love to run. So if I don't run in the morning, I feel weird after. And so I have to do that. I have two kids, two daughters, six-year-old, 11-year-old, and they're endlessly curious. And again, I think what I feel very strongly about is that it's my job to show them you can build crazy things. You can have an idea.

39:53 You can realize that idea. And even though the reality is not as glamorous as that, to try and set that example. So, yeah, it's kind of, that's my day. I feel very fortunate. I love that. And I'm sure they, like, absorb so much from, like, seeing you work and, like, you know, just tackling problems, thinking about it. And, yeah, fantastic. Well, thank you so much, Oliver. Thank you for the time. This was fun. Yeah, it was super fun.

Summary

Odyssey is an AI lab focused on developing world models, which are advanced systems that learn about the physical world through visual observations. Their latest model, Odyssey-3, demonstrates significant advancements in robotics and AI, allowing for more generalized learning across various tasks, such as driving and humanoid control, with a focus on physics accuracy.

- Odyssey-3 is a foundation world model that can be applied to multiple applications, including robotics and driverless cars.
- The model learns general knowledge from diverse tasks, improving performance in specific applications like pick-and-place operations.
- Testing in complex environments, such as busy streets in India, revealed the model's ability to handle unexpected obstacles, demonstrating its generalization capabilities.
- Odyssey collaborates with companies like Flexion to enhance humanoid control policies, leveraging their pre-trained models for specific tasks.
- The AI landscape is evolving, with a shift towards using foundation models, allowing companies to build on existing technologies rather than developing everything in-house.
- Data collection for training models has become more accessible, with a growing ecosystem of companies providing physical AI data.
- The company is navigating a compute scarcity crisis, necessitating strategic partnerships with cloud providers and chip manufacturers.
- Future developments will focus on enhancing reasoning capabilities in world models and scaling their parameters to compete with large language models.

Questions Answered

What is Odyssey-3 and its significance?

Odyssey-3 is a foundation world model developed by an AI lab focused on understanding the world through visual observations. It aims to learn physics, dynamics, and human behaviors, making it applicable to various fields like robotics and driverless cars.

How does Odyssey-3 perform in unpredictable real-world scenarios?

While Odyssey-3 is not yet ready for full deployment, it has shown surprising capabilities in handling unexpected situations, such as animals on the road, by applying learned concepts from pre-training.

How does Odyssey-3 compare to existing language models?

Odyssey-3, with 14 billion parameters, outperforms many language models in specific tasks, particularly those related to physical world interactions. However, language models excel in long-term task execution and reasoning, highlighting the need for world models to evolve.

What impact do data companies have on the development of AI models?

Data companies play a crucial role in advancing AI capabilities by providing the necessary data and operational support. Without them, the progress seen in AI models over the past year would likely not have occurred.

How is Odyssey-3 designed to be compatible with different chips?

Odyssey-3 is engineered to be flexible and optimized for various chips, allowing it to leverage specific features of each chip. This adaptability is crucial given the scarcity of compute resources and the need for efficiency.

© transcribe · For agents Built with care and craft by Gokul Rajaram