Transcript
0:00 10 years ago, we were looking at robotics and saying, "We're just going to solve it totally differently. We're also going to solve it by trying to do a single model that can go from pixels to motor control." And everybody thought we were crazy. Our goal is to build the AI layer that can be used by any robot to solve things that basically require human level understanding and human level precision to accomplish. I don't really believe in a single robot is going to win at all. I think there's going to be a broad ecosystem of many different robot types. So that's why we think building the AI layer that can power all of them is actually the most important way to bring impact into this space and to accelerate it.
0:37 >> I'm looking right now at humanoids and there's sort of a fork in the road right now. There's these companies that are talking about, hey, let's deploy our systems right now. Then there are other companies that are like, we're going to wait until these systems are AGI to really commercialize the product. >> You don't need to wait until you have full AGI in the physical world to make it valuable. Our Gemini robotics model is getting used in robotic applications. Our partner at Boston Dynamics is now launched this with the spot robot. I hope that we can bring a lot of value before full AGI in the physical world is there, but the intention is to continue pushing it all the way until it's there cuz that's when the opportunity explodes. Do you see this work that you're doing with Gemini Robotics as leading to some of these innovations that will sort of like continue to form existing systems beyond like big AGI models?
1:40 Hello and welcome to another episode of Automated. My name is Brian Heer. I am the managing editor at the Association for Advancing Automation. Today we have someone on the show you have been asking for since she stepped into her current role back in May of this year. Carolina Prada is the VP of robotics for Google DeepMind. She has a ton of insight to share around the company's work in physical AI through its Gemini robotics models. Karolina and Google are bringing some really novel approaches to the table. I think you're going to get a lot out of that as well as her personal journey from search to robotics. Thank you so much to her and to DeepMind for setting up that conversation. Thanks to you as always for tuning in. Don't forget to like, subscribe, review, and just generally juice our algorithms and check out our newsletter and news site over at automated.fm.
2:36 Automated's next happy hour and live podcast recording will be in the Bay Area on Tuesday, September 22nd. We will be at Circuit Launch in Mountain View. Thanks to our friends at Silicon Valley Robotics for helping to set that up. More information and a link to sign up in the show notes. Now, please enjoy this episode of Automated. Vision and AI are changing what's possible in industrial automation. From quality inspection and predictive maintenance to robotics, logistics, medical devices, aerospace, and manufacturing. At the advanced vision and AI conference, you'll hear from industry leaders, see real world AI applications in action, explore the latest in machine vision, computer vision, sensors, edge computing, and automation technology. You'll also be able to connect with the people who are shaping what comes next. Whether you're building, buying, or implementing these systems, this is a chance to stay ahead of the curve and see how vision and AI are moving from promise to practical impact. Learn more and make sure to buy your ticket early at the Advanced Vision and AI conference September 23rd and 24th in Santa Clara.
3:47 How does one get into how does one start studying electrical engineering? Why why was that specifically of interest to you? >> Hi Brian, thanks for having me. well, I I I knew pretty early on, like middle school and high school, that I liked math and physics. So, actually, I went to school first for mechanical engineering and then once I was there and I started seeing all the people playing around with circuits and I was like, "Oh, that seems way cooler." So, I ended up switching to electronic engineering. But I knew generally that I wanted to be in engineering that I I've always wanted to I always thought thought about robotics from cartoons as like the coolest thing to build when I grew up. And so that actually motivated me to go into the into that path. And then once you start studying there, there's a million paths to go into, right? So >> that that is a very familiar part of the story too. you know of of just I think it's one just oh yeah like that oh that's actually like feasible and now it like is it is feasible you know as I've seen there's just been so many people that I've spoken to over the years who have studied something like you know maybe a little bit broader you know maybe just some kind of engineering and then once it got to the point where they're like oh there's actually like this is this is like something that I could do for a living there is some infrastructure here then they actually you know do do start studying that. So when when you were in your undergrad, when you were at Washington State, was it specifically in robotics?
5:21 So I was I did my undergrad in electrical and computer engineering at Washington State and then I knew I wanted to go into grad school to get more specialized and in grad school I actually went for control systems because again my intention was to build a career in robotics and then while I was doing control systems I started doing digital signal processing as well you know a lot of those other areas and at that time speech recognition was going through this really hot transition into like be more focused on learning based. It used to be very much like a grammar based approach was what most people would use. People don't think don't remember this now but back then just speech didn't work at all. Right? And so I thought for my PhD I would actually take a diversion and go into machine learning for speech recognition. It was just an area that is was getting transformed. it was also an electrical and computer engineering discipline and so that's how I ended up doing speech recognition in my PhD and I work with one of the grades Fred Jelene was my adviser and he transformed the field so that was a really really great way to get introduced into machine learning and deep learning but it's not the I went after it because it was it felt like a place that was going through a transformation and that it was combining a lot of things that I was learning on signal processing control systems and machine learning all together.
6:50 >> Yeah, that's another through line that I've I've noticed as well. Like there are a lot of people now who are really prominent specifically in the in the robotics field who like again didn't study robotics, but it seems like there are so many other especially technical or mathematical fields that just end up being transferable in one way or another to robotics. Yeah, I mean it's all quite similar like if you're learning if you're doing machine learning regardless of the application is all about learning patterns and the algorithms that you use in order to extract information out of data and so speech recognition went through it and language models actually were the first ones that went through very big transformations and then you see how vision then followed and then now robotics is following but it's all the same underlying techniques. You said you use the word diversion, but like we're talking about I mean that's a significant diversion here, right? I mean we're talking about like 2000 2006, you know, up until I I guess I would say maybe what your time at in video like like 2017. So like >> you know maybe about like a decade or so or so that you were in speech.
8:02 >> Yeah, I did about seven years in speech. was my PhD. Well, after my PhD, I did my PhD on at Johns Hopkins on speech recognition and then I came as an intern to Google and then once I came as an intern and and I also interned in other places but I fell in love with Google pretty early on and I joined at a pretty fun time because back then again speech was not a product it was just a research area and then as we went through this transformation into making it deep learning based it all of a sudden became something that could understand anyone speaking open vocabu vabulary like back back in those days you would do closed vocabulary speech recognition. So you would have a list of states and then you mentioned the state and then a list of stores and then you mentioned the stores and then all of a sudden we were able to just listen in a conversation and understand a 5-second query with voice and that was pretty exciting. So then once we launched speech recognition in many Google surfaces we launched it on voice search so like the microphone now which then went in chrome it went on Android it went on Google home then I was ready for a new challenge and then I went on to Nvidia to do self-driving cars and it was essentially applying the same kind of techniques but for vision now so I was leading some of the vision teams there that were doing like lane detection and sign detection and just all of the different components that you have in your perception system for a self-driving car.
9:30 >> It seems to me that you've got a knack for being in the right place at the right time. you know, because like one of the things that that that you like one of the additional places that cuz it was just an internship, but you were you were at Nuance and Nuance was an SRRI. So, you know, Stanford research spin-off built the I think they're owned by Microsoft now, built like the foundational technology for Siri. You know, you're at Nvidia around the time for self-driving. like like how do you put yourself in these positions? Is it how much of it is is is luck?
10:05 Yeah, I mean I'm sure that there's a lot of luck into it, but I think what I what it is is that I'm looking for what area feels like ripe to be disrupted and definitely the speech the speech area was getting quite more significantly more mature and you could feel that it was those same techniques could apply to other fields. So that's when I started looking to go back into robotics. And so when I was looking through robotics, I was like, self-driving feels like it's now ready.
10:33 And then since then, as you can see, self-driving has come a really long way. None of these fields are like fully solved to be clear. But I was certainly like ready for which one feels that is ready to be disrupted. And then I came to robotics about 8 years ago. So I would say it was still a little early, but just in the last 3 four years, it's been significantly disrupted as you've seen again. So yeah, robotics is hard, but and when I say robotics, I really mean just general purpose robotics, like a robot that can do anything you and I can. And that feels like a pinnacle of showing full artificial general intelligence in the physical world. And so that I still think that we got some work to do to to solve it really.
11:17 >> Yeah, this is going to be an abstract question. So, you know, it might be a bit of a tough one, but, you know, you said none of those are are fully solved. Obviously, like robotics, you know, general purpose robotics is far from fully solved. Self-driving is, you know, pretty far from fully solved. Yeah. I mean, I would say speech is definitely not solved. Like, what is it abstractly speaking? Like, what does it mean to be solved?
11:43 >> Yeah, I mean, I would say that I'm comparing it to human performance. So I would say speech recognition today is has improved dramatically and today we rely on auto captioning for all kinds of videos but still if you put a microphone at a distance in a bar full of people or in a cocktail party you're not going to be able to solve it the same way a human can. So there's definitely other >> put me in a bar I'm probably going to have the same problems but I guess like just generally speaking. Yeah.
12:12 >> Yeah. I think humans are still superior in terms of performance when you're putting them in a cocktail party or even anything that is a little bit farther than close microphone range is hard still. or when you have multiple speakers and you have to discern between them. So there's still aspects where humans are still superior. And then same with self-driving. I think self-driving has gone so far. I'm just very impressed. I ride Whimo pretty often and it's always really exciting to see how it handles almost like a human so many situations but still there's still edge cases that humans are much better at reasoning through that the self-driving still has a ways to go and then finally for general purpose robots is incredibly hard because now at least in self-driving you have rules of the road you have lanes you have traffic lights when you're and you're mostly trying to stay away from interaction But when you're talking about general purpose robots is all about interaction.
13:08 You have to interact with a very unstructured world. There's no rules of the road telling you how to behave. You just sort of have to read the room. Again, if you're working on humancentric spaces. So, it really puts together all of these really difficult problems of understanding the physical world, interacting with it in a meaningful way, and then interacting with humans in that physical world and then being able to reason about what it means to solve a problem. So it is really bringing a whole lot of really complex problems into one.
13:39 >> So >> yeah, I mean it sounds like in in all three cases that human standard is sort of the the gold for each of them. Is that fair? >> Yeah, that's fair. That's definitely what I'm trying to achieve is how do we get these machines to be human level performance? >> Yeah, I would say that like self-driving I mean this is you know again like a whole another conversation but like the self-driving is maybe has to be a little bit better than humans, right? I mean, it's and and and in certain ways like maybe maybe robotics do as well because, you know, obviously again the first time there's an accident or the first time, you know, something's like it goes wrong with a self-driving car, even if a human couldn't have corrected for it, like you know, then there's a lot of push back.
14:28 >> Yeah, you're right. I think on the safety the safety bar is quite high for anything that you put out there higher than a human. But yeah, I I meant to say that definitely human performance is already really really high bar for all of these technologies. And then yeah, of course we can wherever we can make it even safer than human. That's obviously something we want to we want to go after. >> Yeah. Yeah. And and we're gonna loop around to to the work that you're doing with Gemini right now. You know, I know like Gazimoff was was a big piece of that too, but it really and I'm wondering to what extent working in the self-driving world really kind of hammered home how important it is to bake some of these safety standards into robots from the outset.
15:10 >> Mhm. Yeah, certainly. I think the the self-driving world has definitely I mean my entire career has shaped how I see solving each one of these problems but self-driving is certainly the closest and is one that I think it still has a ways to go and it it still feels it represents a small part portion of all that is to be solved for general purpose robots. So there's still a bit more challenges which with general purpose robotics.
15:36 >> Yeah. So, so, so let let's talk about that because I'm I'm curious like specifically we were talking earlier about the the transferable skills and and again one of the things that that I find again when I talk to a lot of people in this field is it really comes down to not necessarily the things that you've studied but people's ways of thinking and people's ways of of problem solving like that's the thing often that is transferable and helpful all of these years you know the decade plus that we mentioned that you were in the world of speech, you know, pattern recognition. I'm I'm going down your Google Scholar page and a lot of the cited papers that you have are in speech. what are some of the learnings from that world that were transferable to the robotics work that you're doing right now?
16:26 >> I mean like like you said, some of it is the the approach with which you see a problem and and how you go about solving it. Certainly one of the things that I've learned with each one of them and if you notice in each one of them I was there trying to change the way the current the current community was solving the problem. So even when I started working on speech recognition one of the first things that I introduced was how to use deep learning for keyword spotting which is at the time was done in a totally different way. So I think in most of these cases I try to go at it from how can I learn this from data in a way that feels robust and resilient to noise and also how do I try to solve this without necessarily evaluating against the current techniques but not just trying to do an incremental improvement on them. So it's perfectly okay to sort of start from scratch and remove the existing approach before be before evaluating against it. So in yeah in speech recognition by the the time that I was in that in that team we were rewriting everything. How do you rewrite acoustic models, language models? At the time we even had multiple systems. We have acoustic language and pronunciation models which then we rewrote again to be a single transformer model. So just like that disregard for the status quo and for just rewriting how you solve a problem is something that I saw consistently across my career. And then in self-driving is happening. It it happened again right at the very beginning. We started with many different systems. One for like lane detection and one for sign detection and one for each one of the different things that you thought ahead of time you have to put together. And then a lot of that became just a single perception system that was thinking through all the cameras and all the sensors together in the car. And so like I think that's one theme that I brought into robotics again. And honestly the jo the team that I joined was already thinking that way.
18:28 So I tend to be also connected to teams that are have that mindset of how do we change robotics? How do we transform it? And then I think pretty early on 10 years ago we were looking at robotics and saying we're just going to solve it totally differently. We're also going to solve it by trying to do a single model that can go from pixels to motor control. And everybody thought we were crazy. And over time it has been adopted more and more and today pretty much everyone that does general purpose robotics is adopted a lot of the techniques that we have introduced. So >> when you say we you mean the deep mind team?
19:03 >> The deep mind team. Yes of course. >> Yeah. >> So we pretty early on maybe 7 years ago we introduced reinforcement learning for whole body control for example in simulation and now pretty much every robot that you see dancing, walking is using reinforcement learning. And that was not back then most people were using just MPC to control robots and modeling everything about the robot perfectly. And then same with the manipulation. Manipulation had and still today there's quite a lot of systems that are deployed today. Most systems that are deployed today are using traditional control approaches and traditional perception.
19:40 But the entire industry is trying to really rewrite that into how do you learn from pixels into motor control to handle uncertainty to be able to understand what's happening in the world to be able to do eye hand coordination. This is something that you couldn't do before. And so yeah, we we're basically again trying to rewrite how to do manipulation for general purpose robots. And I don't think this will be the last time. we will continue to do this.
20:10 >> Yeah, it it sounds like to a certain extent you're you're kind of advocating for every so often maybe switching your focus or bringing people in from the outside because like that's an opportunity to bring in a fresh set of eyes. >> Definitely challenging your assumptions is what I'm advocating for. And I ask folks that join the team all the time that I say we're very proud of having a very diverse mix of folks from different backgrounds. And I asked them all to look at what we have and to be completely candid and challenge our assumptions on how they would do it differently. And so it's it's usually really great to see to get fresh perspectives.
20:48 >> Yeah. I mean it's it's nice to hear that like they were thinking similarly cuz like it it must be a little tough to see like you coming through the door on day one and just be like, "All right, well it's time to it's time to to erase the whiteboard and like start over from day one." Is it is it is it like some is it is it there there has to be a degree in which I mean what do they call it sunk cost fallacy like there there has to be a degree in which it's like it's frustrating right it's frustrating to work on a problem for a really long time and then be asked to sort of like rethink it and then start over again.
21:23 >> Yeah I mean I think we're quite used to it by now. we certainly have some things that we have in place like for example we have beliefs of we want to be able to solve robotics with a single model that can do a broad range of tasks and so that is a fundamental assumption that we have on how we solve the problem but the solution like what is the architecture that we use to solve it what is the way that we learn that we feed data into that model there's a lot of open opportunities to challenge everything that we do about that and then even the ch even the assumption of a single model gets challenge every so often because why not, right? If you're challenging all your other assumptions, that also gets challenged. But for the most part, I think that has been a pretty consistent theme within the team is why don't we build a system that can reason about its environment, reason about itself and then reason about the task and put it all together to understand whether it is done or not.
22:19 That continues to be a theme that has been consistent in the team of the main problem we want to solve. Yeah. I do you feel like across the industry that there's not a consensus but that there's starting to be a consensus around single model because there's definitely still some debate there. >> Yeah. I mean I think there is significantly more attention to how do you solve general purpose robotics with a single foundation model that is now something that the community has adopted as the way forward to really open the door for many more applications for robotics. I think there's still a lot of debate about should you have multiple systems like a system zero that does very high frequency control a system one and then a system two and even within our own team we actually have all of those systems right even so you still think that one of the key ideas that you want to bring forward is that there needs to be a system that was really understanding what is happening when you're interacting with that object and whether that is whether you're moving and progressing in the right ction and how to respond to changes and that is what I mean by a single model but we really have multiple models in our family of solutions right so I'm happy to go into that if you like >> let let's cut right into that right now because when you know Gemini comes out and there are like effectively three different flavors of Gemini 2 there's a main version but you know one of the things I think maybe you're talking about right now is the edge version of it that's meant to run on the machines themselves without the cloud connection And it and I assume that that requires like maybe like a significantly smaller model or a significantly smaller data set at least. So definitely the way to think about Gemini Robotics is a family of models and the we always have running in the in the when you're doing say a 30 minute sequence of the robot doing a task. Say the robot is organizing your office that robot is essentially having a high level reasoning layer. It's just the orchestrator. This is the Gemini Robotics model. It actually understands the environment. It understands what it means to organize the office. Maybe you show it a picture of how your office was organized before and now it has to fix it to that. So it tracks the progress towards that. So it will put the books in the right place and then try to where to put the chairs and what what the progress is towards that goal and it's actually able to talk to you. So that's the reasoning model.
24:43 That's Gina Robotics And then that model operates with an action model which is Gemini robotics the VA. And this action model translates the visual input into the motor commands that the robot executes. So if the robot is putting away the books, the message to that model would be put away the blue book into this bookshelf second shelf on the right. So it's pretty specific about what it wants it to do. And then the action model translates that into the joints moving to grab the object and to move it and put it over there. it actually executes the actions. That action model is still one model that takes his input natural language and visual input and produces all the joint control for the entire robot. And actually underneath that there might be a system zero. If you have a humanoid you need to balance and there might be a system zero that is a whole body controller that in our case is usually an RL whole body controller. So reinforcement learned that is maintaining stability of the system. So actually when you're running organizing your office you actually have three systems that are in play and then the action model that I explained the VA also can run fully on device. So sometimes you might have a situation where there's like no connectivity or very poor connectivity and you just care about the the reliability of that system even if it's less smart because obviously a model that runs on your device is going to be smaller and it's going to be less smart than the model that can run on cloud. At least that's the case today. maybe it will change maybe in two years we won't need a cloud at all and so that's why we have three models that we offer does that make sense so we start with >> high level we have action models and then we have something that maintains stability of the system >> yeah and and I'm guessing this is another place maybe where your speech background is coming into play when we're talking about so you mentioned which is embodied reasoning and the way the way eyesight described by I think it was you on the blog is embody reasoning is what allows it to bridge the gap between digital intelligence and physical action. So yeah I mean for it to actually like execute on an action effectively.
26:56 >> Yeah the model is keeping this long understanding of what it means to do a task and tracking progress towards that task. It doesn't ever move the robot. is the input is literally the goal and the images and then the output is instructions to the action model to execute this and then that and then that. It's basically doing orchestration the same way that any agent that operates on on your codebase is doing orchestration making different calls is exactly the same.
27:27 It's actually we call it our agent >> is is natural language is that a piece of that? Yeah, I mean the modalities that it can use to communicate with the action model can be changed and we certainly explore with that. So sometimes when you're explaining maybe to your kid how to where to put the toys, you might not even like say it all with natural language. You might show your kid, right? Or you might point to it. So those are other modalities that the agent could decide to use with the action model. but these are much earlier in our in our research. So we don't talk about them that much. The majority of how this model was trained was using natural language. And the reason was simple like there's significant amount of data out there on how to translate complex tasks into a sequence of steps using natural language. So it comes out very naturally as a way to orchestrate a long a long task. But we do believe that we're going to need probably multiple modalities to take this all the way to again human level performance.
28:33 >> Yeah. One of the things so I I saw your talk at Machina in Paris and one of the things that you broke down in natural language. One of the things that you broke down really well is the distinction between traditional hard coding and and what AI is bringing to the table. And I think that that's a really big piece of this, right? is actually having to go in and having to code every single action versus these models which were able to to carry out like not not only fluid action but like multi-step actions.
29:04 >> Yeah, I think today there is a signific like I mentioned in that talk today there's significant automation out there and you can automate pretty much anything you want if you can manipulate the environment and put it the way you need and then you can hardcode the entire sequence, right? But that's obviously not very scalable especially in a world that is constantly changing. And so you can think of the reasoning model as the model that takes a whole lot of world understanding and brings in to this applications and you can now orchestrate any sequence of skills and then the action model is basically generating any skill that you want based on natural language and the two come together to do something pretty general.
29:47 >> Yeah. one of the one of the things I I wanted to back up a little bit because one of the things you said a little bit earlier that I thought was really interesting during your time on speech was that speech was not a product, right? So that it was like a fully that it was fully researched in in all these places. So again, like we're talking prior to to Google Assistant, Google Home, Siri, all these things.
30:13 and certainly that was the state for self-driving and like that's the state of of robotics right now and and and it seems like I mean is that the place in which you are the most comfortable operating? >> Yeah, I mean definitely I I'll tell you that my intention is for sure to get to a world in which these robots are useful and they're doing all kinds of things around us and that's not the case today. Today we're very much in research land.
30:41 But my goal is how do how do we continue to iterate on the solution such that it can actually get to useful and for it to get to useful it really needs to be able to handle all that diversity that exists that you and I handle really well. And then all of that variability in objects is never seen. The environment is never seen and just reactivity to changes in that environment. That's really what's going to transform it from something that can be built and be left alone for 10 years to do a single task versus something that is you and I experience every single day. So that's that's certainly where I want to get to. It's a really exciting place to be. I think robotics obviously significantly harder when it's combining hardware that is also evolving with AI that is also evolving and humans all in one is is what makes it super exciting area to work on. Yeah, I think it's really interesting again if you draw the parallels between where we're at in robotics right now and where autonomous driving was 10 years ago.
31:47 there were a lot of early startups in there and very few of them have survived. Whimo is one of them. and a lot of that is, you know, because it's it it's a it's a Google company. It's been around, you know, it's it's it's had sort of the resources to continue and and it's and it's built and it's built those really sort of like deep and and complex models. I'm looking right now at humanoids and there are there's sort of a fork in the road right now, right? There's there's these companies that are talking about, hey, let's deploy our systems right now.
32:23 Let's get these robots out, have them do one or two tasks pretty well, you know, move totes, and then there these then there are other companies that are like, "Okay, AGI, we're going to wait until these systems are are are AGI to really commercialize the product." I know that like Google doesn't necessarily have a dog in that race, but like how do you do you see this playing out in a certain way? Is it going to sort of parallel self-driving?
32:54 >> So, well, first I think we certainly care about solving the whole problem. So, we on the our goal is to build the AI layer, right, that can be used by any robot to solve things that basically require human level understanding and human level precision to to accomplish. So, that's our goal. I think robotics is one of those situations where you're we've envisioned having a really broad set of different robot types that are operating in different environments are doing different tasks. I don't really believe in a single robot is going to win at all. I think there's going to be a broad ecosystem of many different robot types. So that's why we think building the AI layer that can power all of them is actually the most important way to bring impact into this space and to accelerate it. having said that I think you don't need to wait until you have full AGI in the physical world to make it valuable. I think and we're already seeing this we're already putting out for example our agent systems like our Gemini robotics model is getting used in robotic applications to combine existing skills with reasoning so that you can do more complex behaviors. So one example is our partner Boston Dynamics is now launched this with the spot robot and so they use it for as part of their inspection routine and in addition to build using their their regular autonomy they now combine it with a agent to understand what's happening to read instruments to decide what to do next and these are things that they couldn't do before because you had to pre-program it so I think like that there's going to be a whole lot of different applications that are going to start showing up in many different spaces This probably in logistics and in manufacturing it will come sooner because these are more structured spaces where the safety problem is a little bit more constrained but similar to self-driving how they started I don't know if you remember they started in I think >> Pittsburgh >> Pittsburgh and then they started in and then eventually they got to San Francisco and even in San Francisco they haven't gotten to the full Bay Area so there will just be stages right of difficulty that they will we will be posing for ourselves And I hope that we can bring a lot of value before full AGI in the physical world is there. but the intention is to continue pushing it all the way until it's there because that's when the opportunity explodes in terms of applications.
35:14 >> Yeah. And that that that's a really good that's a good point and that's a good example and I think like that works specifically with spot really well because we're talking about systems that are like already deployed, right? Boston Dynamics did a really good job of getting those spots out there without AGI and now you can sort of add this layer. Are you do you see this work that you're doing with Gemini Robotics broadly speaking as you know maybe leading to some of these innovations that will sort of like continue to form existing systems beyond again these sort of like big AGI models.
35:52 Yeah, I think we we see it as there is going to be value to be brought immediately with existing systems that are already deployed where you're combining these agents as a way to bring reasoning and to bring orchestration and to bring more flexibility on understanding the environment and the task and then the action models will start also bringing value in different in different spaces and that is our intention is to actually take existing systems augment them but at the same time we have other bets where we're like we're going to do something that there's no robotic deployments out there today because they couldn't operate in such a complex environment and so anything that has to do with humanentric spaces you really don't have solutions today with general purpose robots that can operate there and we care about those too but we hope that we learned a lot from deployment in these early systems as well >> how I mean I I I don't know this is something so you know I don't know this is something that you've disclosed How are are you able to say how large the team is? The Deep Mind Robotics team, >> I don't think they let us say that, but it's certainly like it's certainly a small team compared to like the full GDM team. But one thing to consider is like when we're solving general purpose robotics, we are also participating and deeply integrated with all of the other Gemini teams. So we work with Gemini and we work with Omni at different stages to shape those models. So even though even if our model is a small our team is a smaller portion of GDM, it certainly leverages and contributes and deeply embeds with all the other teams to be able to shape those models as well.
37:31 >> Yeah. Yeah. And that was something that I really I really started kind of realizing a couple of years ago too was that like it it's clear now in hindsight as I talk to people who have continued you know to make their way up in the world of you know deep mind robotics or or in in intrinsic or you know any sort of like anything on the AI side. It's said it seems like every bet that Google has made in robotics over the years whether it's some like you know some of those acquisitions or everyday robots like has built the foundation and Whimo obviously has built the foundation and like and I'm seeing like all these people like you know I've got like Keep on behind me over here like all all of these people who like came in through these acquisitions in in a lot of cases who are like still there and are still like working with the team.
38:26 >> Yeah, we have some actually quite a bit of people that have been here the entire history. There is some people that are here that were here before I was even part of the team. So, it's exciting to see how deep it goes, the history of the team and then how much we've learned through each one of those transitions. I think in many of those transitions it felt like it was maybe too early to do product and I think this time we're like really taking it seriously to solve the foundations. so like the AI first layer and we're partnering with different companies in order to evaluate that in different applications bring those learning back and continue to push the technology forward.
39:09 >> Yeah. part part of the reason why I I asked about the size of the team is because, you know, I'm I'm curious when we're talking about solving like the biggest problem or solving AGI or at least like these really like robust models and then we're talking about solving some of these sort of you know smaller things that can kind of almost be like patched on to existing systems whether these are complimentary to one another and whether the same people are working on both sides.
39:38 So we only have basically the three models that we put out as three efforts and we don't create separate solutions for each application that I'm talking about. Like even this bot application is simply an application that captures a model that does a lot more than that. So we one thing that we avoid is trying to create like a team per application or per solution and instead we bring applications as benchmarks that help us guide the overall system. And so that enables us to stay nimble, build single models that can do many different tasks but then track performance across multiple applications as they evolve.
40:21 And we leverage of course the deep collaboration that we have with Gemini and sometimes we just get a huge push up just because Gemini fundamentally improved and so that immediately improves all of robotics and other times we are bringing important applications of robotics into Gemini that then boost Gemini's vision capabilities and so it is a cycle that helps us stay focused on a few single models that have broad range of capabilities. I I know, you know, like I I alluded to one one of the reasons why bots dynamics with spot makes a lot of sense again because like there's so many deployed out there.
41:00 I'm I'm curious and and I don't think I've heard you or anybody really talk about this and if this is something that you do talk about, but why Appronic was the company, you know, why that was a fit, why that made sense for this like really strong ongoing partnership that the the two companies have. Yeah, I mean we have multiple as you know we have multiple partners. We have Atronic, Boston Dynamics, Agile Robots and then these are some partners that we work with the deepest but then we also have a broad range of other trusted testers.
41:30 >> I look at the Gemini blog like Gemini 2 blog and like you know Apollo's all over that. So like it's it is a very like it's a strong partnership. >> Yeah. I mean I think it was a certainly for for Gemini Robotics 2 and for each one of these releases that we do we try to capture just what has we always multi having multiple research threads going at all times and we think about each one of these releases as checkpoints along the way. So we don't want to wait as you said to get all the way to AGI to show our progress and so simply the electronic partnership was one that we started earliest and therefore had all of this work around whole body general actions going and it was a great time to show what we have accomplished with them and so that's why it's all over it but it's certainly one that has helped us earliest sort of explore that question of can we get these models to really understand not only things on a table because Even even though that's actually still quite complex, you know, understanding everything that can be done in a table, but we wanted to really understand whether the models could really get to understanding an entire room and understand especially 3D reorientation and understand how their position with respect to objects and that is something that the electronic partnership help us evaluate pretty early on. so that's why it's all over. I think that we are what we look on partners is partners that can help us that can complement what we're doing. So we're very much focused on the models and they can very much be focused on the evolution of the hardware and that can work closely with us because we believe I'm sure well I imagine everybody believes that the robots that we have today are going to evolve dramatically over the next two years. So I think what is most important is that you have a strong team that you can work with to evolve both the hardware and the AI together so that you can get to the best possible point in a couple of years time. And so yeah along the way we're very excited with and very thankful to our to all our partners for what we be able to do and we'll you should expect that we'll continue to show the new partners as well in the in the follow-up releases. Yeah, it's interesting because I feel like that the two ways you know I mentioned this sort of like big bite of the apple that boiling the ocean as they say in in in your neck of the woods of like you know do do we get these things out there or do we solve AGI and then the other piece of that is like big part of the reason why we're seeing these like huge valuations and huge investments is because you know at least a handful of these companies are really trying to go as full stack as as possible. you know including the AI, the software, everything else. but then there are a lot of other companies that that are doing AI but are also partnering with you at the same time.
44:21 and and you alluded to this a bit. I I'm curious when you talk about that partnership, you know, how much of a two-way street is it as far as the hardware is concerned? Like how direct is that conversation? and like ultimately abstractly speaking like how much you influence sort of some of the hardware decisions that they make. >> Yeah. I mean in general the reason that we went together towards a partnership was how do we help accelerate each other and the field that's that's the mindset that we come into it and so we certainly know that we're bringing different parts of the problem into it. And I think we're very open to get feedback in both directions. Right. So we and this matters because in many times you can make your problem significantly harder if you have the wrong sensing modalities or the wrong cameras, the wrong actuators. All of this actually contributes to how hard it is to solve the problem. So we do send feedback back and forth and they also send us feedback on the performance of the models, the applications that they care about because they each each one of our partners has made partnerships with with potential customers, right? And so they know deeply how they want to deploy these models and they bring that feedback back to us. So it actually goes both ways and I think it's really important to do that at this stage. It is still pretty early for humanoid >> for humanoid general purpose robots. So you want to be able to evolve it together.
45:47 I think one of the big points in favor of large models, one of the best arguments that I've seen in favor of large models is I've spoken to a number of physical AI companies that have said, "Oh, we're noticing now like much to our surprise that it's actually useful to train the system on this completely unrelated task and that will inform the job that we're trying to to do here. Is are you finding that to some some extent that like some unrelated actions are actually helping unrelated actions?
46:25 >> So I mean we definitely believe that solving the problem in a general way while it's harder it actually leads to discerning intelligence about what it means to solve a particular part of the task. So to give you an example, if you're actually doing say logistic tasks and all you do in logistic tasks is objects that are red, let's just say, and all of your data is red and all of a sudden you put this robot in a totally different scene, the objects are exactly the same, but they're all blue. I think unless you have two things. Unless you have diversity on your data that involves variety of environments which means different colors and different objects and you have a backbone that understands understands the difference between different colors, you are going to memorize the color as as an associated trait of an object. And so it is going to be a model that is only going to predict to pick up the one red object that is in that place. so that is an exaggeration but it's a lot of what happens right if you it's it's the same as probably I imagine humans will be a lot better than this but if humans grow in only a particular type of world and then all of a sudden you put them in a moon with the very different objects they will be confused and it will take them some time to adjust and so robots are the extreme version of that if they don't see variety they can't discern what matters and what doesn't matter when it comes to doing a task and so I think That relates to what you're mentioning it that what you're mentioning which is does learning a broad range of tasks make me better at learning one task I we believe so we believe so because it enables you to be able to reason about what's different about this task and how I can solve it in under different conditions >> and like it teaches you about physics and other sort of broadly important things like that.
48:13 >> Precisely. >> Yeah. So we're we're at time and and we can close on this. So, I'm obviously going to ask you like a big broad question as we're closing, but this this kind of relates to the the Appronic conversation to a certain extent because around the same time that Gemini 2 was announced, they announced robot park, which I found out was something that apparently Jeff Cardinus, he had the name Robot Park since childhood and finally was able to to use it for this huge like 90,000 foot robot training facility like robots in the real world collecting data. It seems like a lot of physical AI companies right now have moved in that direction like brute forcing it, getting as many out in the world. There's still a lot of companies who are convinced it's going to largely be video for for pre-training. I'm seeing more and more sort of teleresence egocentric teleresence start start to pop up.
49:10 do you have, you know, Google being or DeepMind having been ahead of a lot of these trends, I mean, do do you have a sense that one form of data training when it comes to like really getting this flywheel started is going to be more valuable than any other? Yeah, I mean this is something that we've spoken out about for a while now is like we certainly believe that for robotics to be scalable and to be operating in every humanentric space the same way you and I can they need to have the ability to learn from very rich large scale data and certainly robot data is the most accurate one if you want to map a robot to be able to to behave in a certain way but it's not scalable it it only scales with number of robots and number of hours in the And so we've always had multiple threads going including learning from simulation, learning from video human data, robots learning from their own mistakes. And this is something that the entire industry now of course has adopted. And I think that honestly there's still like research breakthroughs to be done to really properly absorb information this the same way humans can. but we're all making significant strides in that direction. So I hope that certainly the goal at the end of the day is one where robots learn the majority of what they know from observing videos and then they are able to practice a little bit to adjust it to their own body and be able to improve as they continue to do their work. So that's the that's where we all want to go. I think that there's still some breakthroughs to get there.
50:48 >> Well, Karolina, it's been an absolute pleasure. Thank you so much. >> Thank you so much. Thanks so much to Karolina and Google DeepMind for that great conversation. Thanks to you for tuning in. Please like, subscribe, share, review, and check out the rest of our work over at automated.fm. Automated's next happy hour and live podcast recording will be in the Bay Area. On Tuesday, September 22nd, we will be at Circuit Launch in Mountain View. Thank you to our friends over at Silicon Valley Robotics for helping to set that up. more information and a link to sign up in the show notes. Stick around and we will see you back here very soon for another episode of Automated.