Introducing Gemini Omni

0:00 NICOLE BRICHTOVA: There are so many different kinds of edits that you can do, and I think we weren't sure what exactly the model was going to pick up on. But it actually is very versatile. We have an amazing group of people who have been making demos for the past couple of weeks, and I've been really blown away with all the kinds of things that they have been able to do, of, like, turning themselves into crochet dolls, and liquefying things.

0:21 And those things-- and they just work, and it's all through prompting, which is pretty incredible. LOGAN KILPATRICK: Hey, everyone. Welcome back to "Release Notes." My name is Logan Kilpatrick. I'm on the Google DeepMind team. Today, I'm joined by Nicole, Dumi, Gabe, Shlomi. We're talking about the new Omni model-- Gemini Omni. So maybe one of you want to sort of kick us off with the high level news about the model. NICOLE BRICHTOVA: We're really excited to introduce Gemini Omni.

0:52 I think there's two things that people should take away from this model. One is we're basically bringing Nano Banana to video. So we have a really great video generation model, but it especially shines at video editing. And the second thing, the reason we're calling it Omni, is it's really the next step towards the journey of making Gemini fully multimodal in and multimodal out. So you can take in image, video, audio input, references, and, right now, good video as output.

1:20 And then in the future, we will be adding more modalities as a way to make Gemini fully multimodal in and out. LOGAN KILPATRICK: I love it. I feel like we've been telling the Gemini multimodal input story. It was sort of a foundational part of the original Gemini models. And then, obviously, the whole suite from Nano Banana to Veo to some of the music and audio models, like state-of-the-art across the rest of the gen media portfolio. So it's exciting to see it all come together.

1:45 Maybe we look at demos and see, and we can do some live reactions. NICOLE BRICHTOVA: Let's go to some examples. Let's play this one. So this is actually a really nice example of bringing together the Gemini world knowledge media generation. And so this is literally asking the model to generate the alphabet and then examples of an object for each letter of the alphabet.

2:17 LOGAN KILPATRICK: Nice. And how aware is the model of its own-- because I imagine there's a time constraint of how much generation the model can do. Is it aware of its own time? Is the speed sequence, do we have to prompt the model to be make sure to do it fast because you only have 8 seconds to generate? Or how much of that context is-- SHLOMI FRUCHTER: I think that's one of the things that really improved on, is that the model has an ability to create very fast, potentially sequences.

2:46 So unlike previous generations of our models, the control over the time and being able to tell a story is much better. This is a bit of an extreme kind of situation where you want to create very fast frames that go one after the other. Each one has different content, and there is a specific order, so the model is very aware temporally where everything should go. And basically, when the user asks for a relatively kind of complex sequence, then the model is able to reason about it and create the sequence correctly.

3:14 DUMITRU ERHAN: To your question about do you have to prompt it, you can, but you don't have to. It's pretty flexible in that sense. You can just say, well, I do it in ten seconds, and it will do it. But you can also specify, well, I want it to be slow mo, whatever it is that you want it to. So it's pretty controllable in that sense. GABE BARTH-MARON: Part of that comes from the conditioning. So there's this text conditioning that goes into the model.

3:32 Should it be fast paced? Should it be really slow? But then another part of it is just from the model itself. It really looks to find these correlations between what you're asking for and how the video should look. LOGAN KILPATRICK: Yeah, I love it. There's an AI video speed. That first example is like cool to see that it is a lot faster paced than what you would normally get out an AI video generator, which is cool.

3:52 NICOLE BRICHTOVA: And this one's very fast paced, and we prompted it to be very fast paced to fit the alphabet in. But we have another example, which goes into the editing capabilities and also the anything in, because we're using an image as input and a video as input. So let's play a video of this woman talking. - Well, I don't know what it is, but today is going to be a good day. I can feel it.

4:15 NICOLE BRICHTOVA: This is Irina, one of our amazing creators on the Greenfield team. And then the prompt is basically turn me into this animal. - Well, I don't know what it is, but today is going to be a good day. I can feel it. LOGAN KILPATRICK: That's awesome. Just to draw it back to folks who have some familiarity with the Veo models, this is a combination of reference to video audio. And the part that wasn't possible before is the audio consistency.

4:44 Because you could do editing like that with previous Veo generations, right with reference to Veo or no? Or is that not possible? NICOLE BRICHTOVA: You could do a reference to video, where the reference was an image or an audio clip. But video editing is a net new thing that we're bringing with Omni. And that's why we like to use the reference it's kind of Nano Banana for video, because we're basically combining everything that we had in Veo and also Omni, and we're adding video editing.

5:12 And it's really nice because with Gemini in the loop again, because we're building on Gemini, you can use fairly simple language. Literally, turn me into this animal. There's nothing that complex in the prompt, and then the model just does it. And then it keeps the voice the same and it keeps the speech the same, which is really nice. SHLOMI FRUCHTER: Yeah, we pretty much build them all from the ground up to be able to understand different modalities, so in this case, the image and the video.

5:35 So it can actually know what information it has to take from the source or, in this case, the reference video which a person is talking and recontextualize it. In this case, it's an animal talking. But I think the small movements, the expressions of the animal kind of transfer from the original person. And I think, really, those nuances make it look better. GABE BARTH-MARON: Yeah, I think that's a really important point, is that while you could certainly do reference to video, that was the capability that we almost layered into it.

6:04 Whereas with Omni, it's really a step change. It's like we started from the foundation. We had to rethink how to build this model from the ground up. And we're really happy with the results because of that. DUMITRU ERHAN: Maybe pragmatically also, like it actually does improve the quality-- improves with more references that you put in of the same subject. So you can stuff in more images of yourself and more whatever it is, and it will build a better model-- world model if you want to use the term of art.

6:33 And it will just be a better outcome. SHLOMI FRUCHTER: We can also talk a bit more about the way to provide them all with your references. So we have the Avatar workflow, which we'll be talking about more, that allows you to just one time take a photo of yourself, and speak, and get the sample, and then use it across different generations. But also, it can bring 3 or 4 or even more-- up to, I think, about 7-- other references of yours and voice references.

7:03 And again, the more information that the model has, the more it can recreate and recontextualize the presence of this specific person or identity in a new scene. And I think what we've seen is that it's also not just the number of references, but also what information the references have. So, for example, if I take a few photos of myself from different angles, the model can build a better understanding of the 3D structure of the face. And then, if you walk around, then it will actually look more like you, right?

7:32 If you only have a front photo of yourself, then when you turn around, it might look a different person, because the model has no idea how you look from the side. So it's also about that. GABE BARTH-MARON: And it's important to point out that these references don't have to be the same thing or person. So you can have two people. Or in this case, we saw there was a wolf, but you could have even multiple animals, whatever you wanted.

7:52 And then in your prompt, you just point back to the things that you want to incorporate into the video, and it will just intuitively just figure out what to do with those. LOGAN KILPATRICK: Yeah, that's awesome. Let's look at some more demos. I feel like we've got lots of cool ones to look through. NICOLE BRICHTOVA: We have an image of a violin on this background which is just grass, and then we have a woman playing a violin.

8:10 And the first prompt is basically take this violinist and transport her into this environment. So let's play that. Our next prompt is make the violin invisible and then keep everything else the same, right? So let's see what happens.

8:42 LOGAN KILPATRICK: That's awesome. Wow. NICOLE BRICHTOVA: It's really awesome because it keeps everything else the same. She looks the same. She's still playing, the music is the same, and then the violin just disappears and you see what happens. And then this next one is basically change the perspective. So you can see how this is useful for many different things, and it would have been really hard to do. And you can also see how the model just is an understanding of perspective and what happens when you change camera angles.

9:03 So let's play this one. I do not play the violin, but I am told this looks right. LOGAN KILPATRICK: Yeah for sure. That's awesome. Actually, really quickly, on multi-turn, I think one of the pieces of early feedback from the original Nano Banana models was sort like when you do multi-turn, sometimes the model actually wouldn't make the edits. We see that feedback all the time from people, and it's gotten progressively better.

9:36 And I'm curious just to level set people's expectation as they go in and try this model. Is it pretty consistent it'll make the changes that you want, or does it still have that odd artifact that Nano Banana has sometimes, where you'll ask for it and it'll end up just giving you the same image back without anything really changed? NICOLE BRICHTOVA: It's less that. I think the main difference is between Nano Banana and Omni here is obviously the generations take a lot longer.

9:59 So Nano Banana was super snappy. You have ten seconds and then you get the output. Here, you're probably waiting like 60 seconds plus, sometimes 90. So I think that naturally maybe limits how much you want to do multi-turn. But it definitely-- I think the other thing is that we've seen this with Nano Banana, too. Obviously, the longer you go, maybe the more the instruction following degrades, and that's still true. But we've been getting 2, 3, 4 turns pretty reliably with this model.

10:26 But I think to your point, I think every time we introduce new capabilities like this, sometimes the model is better at following some instructions than others. So please send us feedback. I think we're all on Twitter. If we're not, we should be. And tag us and let us know what you like and don't like about this. LOGAN KILPATRICK: Don't tag me. Tag them. DUMITRU ERHAN: I don't know. Yeah. Maybe one limitation that I have personally observed is that sometimes editing will edit more than you asked for.

10:49 I think it's almost the opposite of what you're asking about. It'll be like-- and that's-- I don't know. We need to dig deeper into how and which way is the problem. But it's possible that if you under specify what your edit is, the model will do more than you thought should be do-- should be done, right? You may be technically correct. It did what you did, but it also did some more. SHLOMI FRUCHTER: When it comes to videos, there's so much happening.

11:13 Ten-second video, actually a lot is happening. And to be able to edit it accurately, even explain what you want, with language is not always very easy. So sometimes you might ask for something to change, but again, it is kind of ill-defined. What's the right thing to do? Editing is not always clear. OK, I want to change this text to something else. That's one thing. But if I want to change the-- make it invisible, there are many ways, for example, like we've seen in this violin, to make it maybe invisible or change the camera position.

11:41 There are multiple ways. So the more you're specifying what you want, I think the better the model can understand and try to implement the edit. But still, of course, it doesn't always get it right, and we're still working on getting it to be more and more accurate. LOGAN KILPATRICK: How much reasoning is actually happening? Is that part of that 90 seconds, is the model actually doing or 60 seconds the model is actually doing a bunch of reasoning similar to the latter Nano Banana models, where that's a core part of how we're getting model performance out of it?

12:10 GABE BARTH-MARON: Yeah, there is quite a bit of reasoning that goes on behind the scenes. I mean, if you think about when you're doing text to video, and you give it a simple prompt, as you said, there's a lot more going on in the video than just like a dog on a skateboard or something like that. Well, where's the dog? What time of the day is it, et cetera? And so the model really needs to create a lot of detail.

12:31 And so it needs to reason about it, just like a person creating a film would reason about, well, what's the story? What's going on? LOGAN KILPATRICK: And that's actually what's happening, is it's reasoning through scene by scene or something like that? Or I don't know how much of this is the secret sauce or how much is actually even visible to users at all, but what happens as it is thinking through what the next 8 or 9 seconds of the video should look like?

12:54 SHLOMI FRUCHTER: If you provide a very short prompt, then there is so much to create to just give you a 10-second video. So a lot of this information is coming from the model thinking through how it should be, maybe split the 10 seconds or longer to connect shorter kind of clips. What should happen in each one, how they should be consistent with each other-- so there is lots of planning around that. And then eventually, the video that you get tries to combine all of this information that are in this planning into a coherent video.

13:30 So if you provide more information, for example, and one thing that I like is that you can actually say I want in specific time, specific seconds for something to happen, then you constrain much more the video. So if you provide more information, the model will try and adhere to what the user wants. And then, if you provide less, then it has to go and be more of a director on your behalf. LOGAN KILPATRICK: Nice. I'm curious why 10 seconds.

13:54 Is it just the model has gotten better and more coherent or architecturally it's set up to do 10 seconds? Or why 10 seconds? Why not-- I know 30 seconds would take a long time to generate. SHLOMI FRUCHTER: With Omni, you can create 10-second videos. That's true. But because of the consistency, that using references, you can easily have the same basically character or characters in a 10 second video. So you can easily create a much longer film.

14:21 That being said, we are definitely planning to extend that in the near future. NICOLE BRICHTOVA: We really tried to focus on making this accessible to consumers with this release. I think that's quite new, because with Veo, we really focused on professional creators and filmmakers. And we're still working with those, obviously, because everything that we've shown you is useful if you're creating content professionally. But we really wanted to make this accessible to consumers. So the fact that can create your avatar, and then you can put yourself in videos and it looks like you, that's kind of that Nano Banana moment for video.

14:51 Shorter videos we also think are more conducive to the consumer use cases, but we are planning on enabling longer videos. And one interesting thing is when we talk to folks who are creating films professionally, they actually prefer to have shorter scenes, because you create a scene, and then you stack those scenes. So the workflow that Shlomi described is actually quite common in the industry. But you're right. We do want to make longer videos, and that's definitely on the roadmap.

15:15 DUMITRU ERHAN: There's also a user experience. Imagine doing a multi-turn edit of a 90-second video. Most users will just not find this very pleasant, but we could do it as a flag planting thing if you want. GABE BARTH-MARON: We'll get there. DUMITRU ERHAN: Yeah, we'll get there. GABE BARTH-MARON: We'll get there. LOGAN KILPATRICK: One of the things, actually, that we haven't talked about, and this is related to use cases and the different audiences and everything they're looking for, maybe we can actually just talk about availability for a second, which is like for people trying to do the Nano Banana or Veo moment, where should they be going for the more professional user who wants to do the 10-second multi, many, many iterations, put together something professional, where should they be going-- availability?

15:53 NICOLE BRICHTOVA: For easy consumer use cases, go to the Gemini app. So it's available to Ultra and Pro and Plus users. LOGAN KILPATRICK: Really? Nice. NICOLE BRICHTOVA: Yes. LOGAN KILPATRICK: And that's for all the TPUs are going. Now it makes sense. NICOLE BRICHTOVA: No, but actually, this is related, right? Because if we do longer video generations, then we can serve it to fewer people. So this is definitely a trade-off that we thought about with this model.

16:13 But yes, go to the Gemini app. For people who have much more intensive workflows and who want to really use a full-on creative suite, go to Flow. You can use Nano Banana to create references, characters. You can give your characters voices so they're kind of consistent throughout the project that you're working on. And you can use Omni inflow for all of those creations. Flow now has some really cool features where they also have a agentic workflow that can suggest you ideas.

16:39 It's very cool, actually. Shout out to the Flow team and Josh-- where an agent can suggest you ideas on what to do. I usually don't have good ideas on what to do, so I just use the agent to tell me what to do there, and that's really fun. And then we're also, I think for the first time, also some shipping this model with YouTube. And so if you go on YouTube Shorts or YouTube Create on YouTube Shorts, basically some videos will be eligible to be remixed with Omni, and then you can go in and edit those.

17:06 LOGAN KILPATRICK: Very cool. Nice. NICOLE BRICHTOVA: And then we will have APIs and other-- LOGAN KILPATRICK: Developer access soon. NICOLE BRICHTOVA: Coming soon. LOGAN KILPATRICK: Fingers crossed. I'm excited. SHLOMI FRUCHTER: The user experience of creating videos is not like-- it's not a solved thing. And it really depends on what the user is trying to do. So I find myself using both to Gemini app for some things when I just want something to create quickly, maybe some meme to sent to someone.

17:28 Maybe I want to take my video and just change something small, and make it funny, and send it to a friend or my family. But then if I want to try and create, it's not like I'm an amazing film-- amateur, amateur. NICOLE BRICHTOVA: We've seen some of your films. SHLOMI FRUCHTER: But yes. But my kids like it. And so, I mean, so it can be professional, but it can also be maybe more of a longer workflow, and you have some idea, and you want to get it out and actually create it.

17:55 And then, I think, there are amazing improvements to Flow and to the user experience there. You can build those characters, and give them a voice, and then use them in different scenes. So the ability to create a much longer form or a video, I think it's just a better workflow for that. LOGAN KILPATRICK: Yeah. SHLOMI FRUCHTER: So yeah, we kind of tried. We believe both those use cases are very different and require different experiences. LOGAN KILPATRICK: What about-- and actually, maybe we go to the next demo, but I'm also curious to plant the seed for multi-human video generation and editing.

18:28 And I'm curious how the model does with character consistency. Is the suggestion two people, it's pretty reasonably good, you get to five, it starts to break down and things get crazier? And I feel like it was the same for Nano Banana. For editing, a group photo was one example that I felt like didn't work on the initial iteration of the model, and you needed to have one person or two people. Same vibe for Omni to start, or can it handle it all?

18:55 NICOLE BRICHTOVA: I've seen pretty impressive examples of two characters, and maybe stretching it to three. Again, you do have to provide quite a few references. So the more image references you provide for each character and the better you get all of their facial angles so that you can capture the movement, the better things get. But two to three, pretty good. I think once you go to a big group photo, and then you try to animate it, it starts to drift much, much faster.

19:19 And even if I use a single headshot of me, and then I try to put myself in a scene, it doesn't quite look like me. But if I do the side profile from both sides and head on, it's so much better. KILPATRICK: And can those references be videos? Could you do, like, you look at the camera, side to side, up, down, like you're-- NICOLE BRICHTOVA: It will be. LOGAN KILPATRICK: It will be? DUMITRU ERHAN: Coming soon.

19:38 LOGAN KILPATRICK: Coming soon? DUMITRU ERHAN: The model can do much more than we're actually shipping. SHLOMI FRUCHTER: Let's say we're going to take us two months at least to ship all of the features. It has a lot in it. NICOLE BRICHTOVA: We packed a lot into a small model. And I think this is actually part of why it's called Gemini Omni Flash. It's actually really impressive because it is a pretty small model. We're packing a lot of features, and we have more coming.

20:01 GABE BARTH-MARON: So we knew this when we were developing the model, but we were developing something that we didn't know how to serve quite yet. And so we're kind of slowly unrolling this as we figure out, well, how do we actually want to let users interact with it or allow users to interact with it? One thing I would say is that the character generation, especially from T50, is quite good for four or five characters. I think the one thing that is an area to improve upon is voice generation.

20:27 So once you go beyond two characters, what you might start seeing is that the same person might be talking-- it kind of gets confused between who should be talking exactly when. And this is obviously an area that we want to improve upon. LOGAN KILPATRICK: I love it. So our podcasting with five people, we're good. NICOLE BRICHTOVA: You're still good. LOGAN KILPATRICK: We're still good now. But if it's only two people, don't do two. It's a matter of time.

20:50 Let's watch some more demos. NICOLE BRICHTOVA: This is another example of just multiturn of impossible worlds that you can have in the palm of your hand. Let's do it. LOGAN KILPATRICK: Oh, wow. I love this. It looks like that little world orb thing from Project GENIE. NICOLE BRICHTOVA: And it's just so much more accurate. We tested all of these against Veo. And you can really see the step change from where we were with Veo, so it's pretty exciting.

21:23 If you go to the next one, this is just celestial world-- it's basically a play on the same idea. SHLOMI FRUCHTER: Five fingers NICOLE BRICHTOVA: Five fingers. LOGAN KILPATRICK: That looks like a good hand. That's a good hand. Very cool. NICOLE BRICHTOVA: And then we have another one. LOGAN KILPATRICK: That's very cool. NICOLE BRICHTOVA: Very timely. SHLOMI FRUCHTER: The text rendering, I think, is really something that we love, that I think it's-- in text rendering, we really feel like it's a step change compared to our previous video models, and it's so useful because text, apparently, is pretty useful.

21:58 So for anything informational, or even if you just want to make some fun message to someone, being able to have text on screen-- GABE BARTH-MARON: Or adding an overlay with video. SHLOMI FRUCHTER: Yeah, it's just really-- and of course, the text-- I think we have some examples that the text is in the right place. You can read it. It looks correct. And I think this is super useful of capability that we are improving. And in a way, that's coming from this Omni approach because the model learns all of modalities, including text, from the ground up.

22:31 So it can actually do much better on text generation thanks to that. LOGAN KILPATRICK: I feel like the Omni model has to be the hardest example of taking all of these different things and bringing them together. And I'm curious, actually, just how we think about prioritizing, what do we want to be good? Was it actually initially we were like, hey, the Nano Banana moment was incredible, we think we could do something really, really interesting for video, and so let's go and try to make sure that use case works really well to begin with, and then keep hill-climbing everything else to try to get quality up?

23:02 Or is it like, you just throw it all together and then do a bunch of evals and play around with it, and you're like, actually, video editing is a thing that works best? DUMITRU ERHAN: We definitely started-- the tagline of the project, which started a while ago, was Nano Banana for video. That was there in the slides ever since the beginning of the project. So this was definitely sort of a north star there. It doesn't mean that we didn't do the other thing where you said, where we just put everything together and hope that it works.

23:27 But it was definitely where we wanted to go to. And we've arrived. But I think it's definitely an open problem of how to put everything together in a way that gets you to that desired outcome because Nana Banana for video, we all feel it. We understand what that means. But actually practically measuring, are we improving towards that goal? It's not super obvious. There's just a lot to video editing that is not just quantified in a single number they can hill climb on.

23:57 GABE BARTH-MARON: For us, execution is really key, and we're researchers, so we incrementally added these things and layered everything on top of each other. And, of course, everything interacts with each other. I think one of the most difficult parts of this particular model was just the number of evaluations that we had to run, especially towards the end when we just had everything going. And so we had editing references, image out, text out, et cetera. We just had to evaluate everything and then just understand what's going on.

24:25 And you see some things regress, some things improve, knowing when the trade-offs are, where those trade-offs are-- it's really challenging. You need a very deep intuition on how these models work and what's going to work at scale and what isn't. SHLOMI FRUCHTER: First, it builds on, I think, long-term research that we've been doing, I think, for a few years, how to combine modalities. And as you've mentioned, this is being a north star for Gemini and our generative media projects to see how we can combine the modalities together.

24:54 And interesting thing is that as you combine different modalities, you have to decide to prioritize which one is more important or how do we weigh them in a way that gets us what we want the model to be able to excel in. And that's something we definitely worked on a lot. The interesting thing is that we see that putting some of the modalities together actually helps it-- different modalities because there is a lot shared, for example, for image generation and video generation.

25:23 There are a lot of similarities of generation of the visual stuff, understanding the visual world. Same goes for music and audio. You can imagine that if you ask for music to be generated in a video, then the model has to know how to create the music to begin with. So we also see that this information helps the model on to do better on videos when it has, for example, audio and music information. So overall, we see that all of those modalities make them all better across the board, of course, with the right adjustments.

25:52 LOGAN KILPATRICK: Historically, we actually talked about for Nano Banana that Kaushik and others had trying to-- the text rendering quality is an overall metric that you can track correlated with quality of the model in general. And I'm curious if that's been true for Omni as this text rendering capability has gotten so good. And then, actually, also the tooling, tool calling capability of the model, and obviously the latter versions of Nano Banana that you can use search out of the box and has, I think, image search, which is really cool and has made that experience powerful.

26:25 And I'm curious how much that's the tool use for the Omni model is something that either works today or is something that you all are pushing towards in the future. GABE BARTH-MARON: Works today, that's for certain, but it's definitely something we're pushing on as well. NICOLE BRICHTOVA: I think what you've seen with Nano Banana kind of generally-- we started with a lot of the fun use cases of edit an image of me and make me a mini figurine.

26:46 And then we went to, actually, you can make an infographic and use it for work presentation and whatnot. Or you can use it to explain a really difficult concept to someone. And I think you're kind of seeing us do that with video now too, where we're starting with a lot of these fun things, but then you're like, oh, it's actually also useful because I take a video on vacation, and somebody walks in, and then maybe I just want to remove them.

27:09 In the past, most people would have never learned the professional tools to be able to do those kinds of things. So I think we're in that stage, and we're seeing the early glimpses of, well, actually, now you have text rendering working. And then, when we add in the grounding and actual real information-- we're also launching kind of Street View grounding with Genie at I/O this year. So you can see how you can ground in a lot of this real information.

27:32 It just makes the models much more useful. It's definitely where we're going. SHLOMI FRUCHTER: I think it connects ultimately to Google's mission of organizing the world's information. And we really see how there are really cool applications that are more on the informational side and less on the entertainment side, which, of course, is interesting as well. But the information that we see were just the tip of what we can do. DUMITRU ERHAN: It works really well despite not having poured super many resources into it.

28:00 So I think that's-- and I'm actually quite excited to see, is there anything else that we haven't discovered? So people will discover the mini figurines or whatever it is that people have done with Nano Banana, I'm sure people will do similar interesting things with Omni when it ships and what they tried. LOGAN KILPATRICK: Those Yeti selfie vlogs or whatever, I'm like, now you can just be an influencer without actually having to go. Just send my avatar out and go explore the world.

28:28 NICOLE BRICHTOVA: This example, actually, is also really cool. I've really loved examples that people have been doing of you take footage of the real world, and then you do something crazy fantastical in that world. So this is an example of a person who's just drawing a circle on a piece of paper, and then you turn it into all these wild things. So this is a circle turning into a black hole. DUMITRU ERHAN: Black hole or DeepMind logo?

28:57 LOGAN KILPATRICK: Wow, that's awesome. SHLOMI FRUCHTER: Don't try that. LOGAN KILPATRICK: Yeah, don't try that at home. SHLOMI FRUCHTER: One thing that we really like about the model is that it can do style-- kind of changing the style of a video, and that's something that we don't have any such data in when we train them. It's very hard to have a video in one style and then a video in another style. That's not something that you can easily find.

29:20 But the model just kind of generalized, and because it has a deep understanding of different types of media and styles and language, and you can just say, OK, I want to have this video in a different style, and I'll edit it too, and then it will just learn to change the style in a way that looks reasonable. So here, for example, we asked basically the model to change the style throughout the video. And you can say the scene remains consistent, but the style just changes.

29:45 NICOLE BRICHTOVA: And I think this is interesting just for video editing in general because there's so many different kinds of edits that you can do. And I think we weren't sure what exactly the model was going to pick up on, but it actually is very versatile. We have an amazing group of people who have been making demos for the past couple of weeks, and I've been really blown away with all the kinds of things that they have been able to do of turning themselves into crochet dolls and liquefying things and those things, and they just work.

30:13 And it's all through prompting, which is pretty incredible. GABE BARTH-MARON: Something I was really impressed with was actually an early iteration of the video right after we recently allowed it to do video edits. I was looking at some evaluation prompts, and I saw this one prompt-- well, most of them kind of asked for these kind of modifications to the scene. But I saw this one prompt that asked for the next scene, and the video just flawlessly executed it.

30:34 It continued the story. There's a woman walking down a corridor, and then it asks for a monster to come out of a door. And then the camera turns around the corner, and it just continued and did it. And that was not something we had ever explicitly trained for. And that's actually something that you can do today, which is pretty cool. LOGAN KILPATRICK: I feel like this is an interesting new form of media that doesn't really exist, which is this have those choose your own adventure books growing up that you read or whatever, and you can do that now with Omni, but in video format and a much more engaging way, which I think is really, really interesting.

31:08 NICOLE BRICHTOVA: I think the way that we generally think about media now is different people learn through different ways. And a lot of people are visual learners. And so I've had a lot of people come up to me after Nano Banana saying, like, oh, my dad is a chemistry professor, and I have no idea what he does. And so I just took all his lectures, fed it into Nano Banana, and got these sketch notes that just visually explain it to me because that's how I learn.

31:29 And it was the first time that I was able to have a conversation with my dad in 20 years about what he does. And it was like a real bonding moment for that family. And so I think, again, with video, some people may understand concept better through video or some concepts are better explained through a video because you need to see the motion and the transition. So one of the really exciting things that I think we have a protein folding example, very on brand, is you can explain things in different styles.

31:54 You can adjust it to different audiences. And so I think one of the things we're all really excited about is just a new tool for learning because a kid was going to learn differently than I'm going to learn about protein folding. And so we have one example here. SPEAKER 1: Proteins start as chains of amino acids. They fold into patterns like the alpha helix and flat sections called beta sheets forming a perfect three-dimensional shape. LOGAN KILPATRICK: I need to extend this video a few times.

32:18 NICOLE BRICHTOVA: We're going to have to-- this is when extension comes? Now you can go deeper. DUMITRU ERHAN: And this is where the promise of longer videos is going to hold, and it's going to be more useful than just arbitrarily generate something very long for no good reason. I think generating longer videos that explain things that are educational, that create bonding moments, I think that'll be very cool. LOGAN KILPATRICK: I also feel like with this educational type of content with style transfer is also really interesting because it's like, oh, I can make it comic book-themed or whatever it is, and you can bring it to life in different ways and not if you don't want this claymation animation style or whatever it is.

32:55 GABE BARTH-MARON: I had a physics teacher that he loved bringing memes and things from my generation into the classroom. And it helped. It made it a lot more engaging. And so you can imagine a teacher bringing videos in and really engaging the students a lot more with those. SHLOMI FRUCHTER: Yeah, I think it's really promising as a tool for educators and teachers to be able to bring a concept, an idea, and they can explain whatever they want, but in a visual way that would really match their audience.

33:27 So it's explain to me like I'm five or explain to me like I like memes. The medium is basically matching the audience, and I think we can go much, much deeper on that. LOGAN KILPATRICK: It's somewhat obvious, but it's also worth-- for folks who have tried to make even this type of basic-- I've done this a bunch for programming stuff, and it's extremely time-- it's like, I want so much to do it because the end product, I think, is great for people.

33:50 It's just super laborious. It's not what brings energy in my life. And I feel like there's so many cases now where I think folks are going to get their hands on Omni and go and make a bunch of this content, which is going to be great for other people, and it just wouldn't have existed otherwise because they don't have the time, means, money, whatever it is, in order to do it. And now you can get a Gemini Pro subscription.

34:12 And you're off to the races and can go and use those, which is really cool. So that's awesome. NICOLE BRICHTOVA: So we have another demo where we basically show any modality in. So we have some images, we have audio references, we have a video. If you remember the sailor from last year's I/O, he got very famous on Twitter. We show him all the time in our review decks. And so we have an example of taking a style of a storybook and then the sailor from last year and his voice, and we've got a new track for you.

34:47 SPEAKER 2: The sea is a wild, untamed might. She commands your awe. LOGAN KILPATRICK: Very cool, I see. I feel like I need more. I'm like, give me another give me another 20, yeah, give me another 20 seconds more. NICOLE BRICHTOVA: We hear that. LOGAN KILPATRICK: And so actually, just for this example, this was the references for this sailor at the sea was the previous-- a screenshot of what it was before? What's the-- NICOLE BRICHTOVA: It was the previous year's video with that audio track-- so it was basically what he said in this video is exactly what he was saying in the original video.

35:22 It was kind of looking at into the sea and into the distance. And then we had a style reference image that went into this. We've got one more example, where you gave us your permission to use-- LOGAN KILPATRICK: Sammy took this picture to use your likeness. I did not know what it was for originally, but I did consent blindly. NICOLE BRICHTOVA: You did consent to your picture being taken. And we have a demo of you at Shoreline.

35:46 LOGAN KILPATRICK: Nice. Gemini. That's pretty uncanny. NICOLE BRICHTOVA: Looks pretty on brand. LOGAN KILPATRICK: That looks pretty on brand, honestly. DUMITRU ERHAN: I mean, I think I can see your next Twitter post. LOGAN KILPATRICK: Yeah, exactly. NICOLE BRICHTOVA: I just have one more video. Let's take a look. LOGAN KILPATRICK: I've been working on training this new model on my likeness and voice. It is Google's new Omni model, and the results are pretty wild. Does it actually sound like me?

36:16 It is hard to tell. NICOLE BRICHTOVA: Does it pass the Logan test? LOGAN KILPATRICK: Yeah, that passes the test. Throw back to my old apartment. I miss it. The first one was very, very good. I think the first one was A+. I think the second one was like, A. SHLOMI FRUCHTER: So in that case, how many images did we use? NICOLE BRICHTOVA: I think we used multiple photos to get-- LOGAN KILPATRICK: Sammy did take some side profile shots.

36:40 NICOLE BRICHTOVA: --you need the side profile shots that actually-- when you move, it kind of looks like you. But I will say, even evaluating yourself is sometimes very hard, especially on your voice because you don't usually listen to yourself speaking. So to Shlomi's point, it's actually really helpful to just see other people on the team, and then you look at if the model is good at reproducing their-- DUMITRU ERHAN: Well, Logan, may be uniquely qualified because he does listen to himself a lot.

37:02 LOGAN KILPATRICK: Unfortunately, I have to hear myself talk. Maybe this is actually a great transition to talk just generally about safety for this model. Obviously, the capability isn't super interesting. Obviously, generating things with people's likeness is-- I'm sure there's many, many challenges product model-wise to handle. So we'd love to talk about, at a high level, what are the things we're thinking about, anything that folks should know as they get their hands on Omni? NICOLE BRICHTOVA: There's obviously a lot of new things that you can do with this model, including edit videos, which includes speech, including using voice and audio references.

37:36 And so we are actually going out pretty conservative when we first released this model. And you might feel like we're blocking a lot of things. And part of the reason is because we really are trying to learn what people will do with this model and just really look at real-world use cases because we have some idea of what people will do, both positive and negative, but we really need to see what happens when we release the model in the real world.

37:58 And so we are taking a pretty conservative approach. For your own voice and your own likeness right now, you can only do it through the avatar flow. And we will monitor the feedback. Please do send us feedback on what you think should be working and isn't right now, and we will adjust as we go, but we really want to learn and get the feedback from users so that we can both kind deploy this responsibly, but also give people creative control over what they want to do, which is the balance that we try to strike every time.

38:23 LOGAN KILPATRICK: I don't think we've mentioned yet the avatar flow, and just like the really quick TLDR for folks who want to go do, actually, with their own likeness, create some sort of video? NICOLE BRICHTOVA: Yeah, so this is also kind of a convenience thing because I think as we've been talking about, in order to capture your likeness, you really do have to take multiple shots of your face. We need an audio clip. And so the avatar flow is a one time setup that you can go through in all of the products that we're launching actually, so YouTube flow and the Gemini app, and basically, you record yourself speaking.

38:54 So you speak out a set of numbers out loud. We capture your visual likeness on camera, and then you basically store your avatar and you can invoke it the next time you come into the product. So both has this sort of safety element, but it also has the convenience element of you just have to do it once, and then you can reuse your avatar every single time that you come into the product. SHLOMI FRUCHTER: I think SynthID and the ability to know if video is basically generated, I think it's really key because we see that the capabilities of models across the board get better and better.

39:23 They look very real. We can have someone's identity very carefully kind of replicated. And that obviously poses a challenge. And I think, ultimately, part of releasing this responsibly is that all of our videos are always embedded with this SynthID, basically, mark, that watermark that says that it is possible to know this video is generated, and then on various surfaces, like Chrome and Gemini app and others, users can check if a video that they have or received was generated by our models.

39:59 And we think that this is basically probably one of the most important ways to know that content is generated. And you should know if you want to trust it or not based on that information. NICOLE BRICHTOVA: And just on that. So we're obviously very proud of SynthID, which is a tool that we've developed, but we also partner with C2PA and we embed C2PA metadata in all of our content so that if people want to check it on other platforms, they also can do that.

40:23 GABE BARTH-MARON: And like SynthID, it persists across edits as well and if people make modifications in post as well. And I just want-- so everyone knows the way to use it is you upload a video to the Gemini app, you can ask, is this an AI-generated video? And it will check that for you. LOGAN KILPATRICK: I love it. Well, this was awesome-- a lot to look forward, a lot to hopefully get the feedback. Y'all are all over the internet.

40:48 Please send us feedback about the models and what doesn't work. Maybe we can also just talk for a second before we close about what comes next. Obviously, Omni Flash, hopefully, will have a Pro model, but anything that y'all are especially excited for in the next iterations of the model? SHLOMI FRUCHTER: Personally, I'm really excited about pushing the envelope on what's possible across quality-wise, the duration of video-- LOGAN KILPATRICK: You say that about everything. SHLOMI FRUCHTER: Everything.

41:15 But specifically, I really would love to see people being able to create to tell stories. I think that's something that we've seen that with our previous models. It was pretty difficult to actually tell a story, that something happens, and then something else happens, and things remain consistent. And we think there is-- we made progress, and we hope to make even more progress towards that. So that can apply to filmmaking, to informational videos. And I'm the most excited about that.

41:46 And I'm personally also excited about being able to have more interactive versions of our models. So we hope to have some of that in the future. GABE BARTH-MARON: I'm really excited about being able to deliver factual and information content. So I think, if you think about the progress of video models over time, as they become better at grounding and these better world models, you go from enabling people to generate things that are really entertaining and engaging videos to things that people can use for learning and information delivery.

42:22 And I think that we're going to be able to really push on that and push the envelope there in the next generation. LOGAN KILPATRICK: I love it. DUMITRU ERHAN: This kind of technology will empower a new generation of YouTubers, I think, of just content creators that will hopefully build interesting kind of educational information or whatever they are things. I'm also actually excited about new emerging capabilities. I don't think we've demoed this here, but we've discovered that the model is pretty good at synchronizing music to video.

42:50 So you can create videos that respond to music-- again, not something that we'd trained for specifically. I'm wondering what other things will people discover this way. What other things is this model capable of? And then maybe we can double down on that in the Pro version, whenever that comes. NICOLE BRICHTOVA: When we talk about these Omni models, I think there are interesting ways in which the modalities can interact that we haven't really, I think, fully discovered yet.

43:19 So you can iterate on a storyboard in images, and you can design your character before you ever go into video generation. And I think we're kind of just scratching the surface on that. And that's going to be a really interesting way, I think, to see people interact with these models. And so I'm excited about what's to come on that over the coming months. LOGAN KILPATRICK: I'm also excited. This is going to be great. Well, thank all of you for taking the time to sit down for building a cool model.

43:41 All the other teams across Google who sort of made this launch possible-- I think there's probably like hundreds and hundreds of people in collaboration from serving to training to research to all the product work to bring all these things to life. So yeah, a great Google team effort. Thank you all for sitting down. And thanks, everyone, for tuning in to this episode of Release Notes. We'll see you in the next one.