transcribe

Why FLUX 3 Trains on Images, Video and Sound — Andreas Blattmann & Dustin Podell, Black Forest Labs

Latent Space · 1h 38m · transcribed 4d ago
More from Latent Space Business
𝕏 Share ▶ YouTube 📥 PDF 🤖 .md

Section Insights

# 0:00

Introduction to Black Forest Labs

Who are the founders of Black Forest Labs and what is their background?

The founders, Andreas Blackman (Andy) and Dustin Podell, have a rich history in AI and image generation, particularly with stable diffusion. Andy co-invented latent diffusion, while Dustin transitioned from a freelance web developer to a model and research lead at Stability AI before co-founding Black Forest Labs.

  • Black Forest Labs was formed by key figures from Stability AI.
  • The founders have diverse backgrounds in academia and open-source development.
  • Latent diffusion technology is central to their work in image generation.
# 14:02

Team Diversity and Backgrounds

How does the diversity of backgrounds contribute to the strength of Black Forest Labs?

The founding team comprises individuals from various fields, including hardcore academia and the open-source community, which enhances their collaborative strength and innovation.

  • Diverse backgrounds foster creativity and innovation.
  • Collaboration between academia and open-source communities is beneficial.
  • A shared history among team members strengthens their bond.
# 28:05

Advancements in SDXL

What are the key improvements introduced with SDXL?

SDXL introduced significant enhancements over previous models, including multi-aspect training, improved aesthetics, and the ability to handle various image resolutions, marking a step forward in the evolution of stable diffusion models.

  • SDXL allows for multi-aspect image generation.
  • It incorporates large-scale training techniques.
  • The model's enhancements improve overall image quality and versatility.
# 42:08

Understanding Diffusion Models

What is the significance of flow matching in diffusion models?

Flow matching is a mathematical approach that allows models to translate between different probability distributions, making them more scalable and easier to work with, thus enhancing performance in generating images.

  • Flow matching simplifies the translation between distributions.
  • It allows for more scalable and efficient model development.
  • Understanding these mathematical principles is crucial for advancing AI models.
# 56:11

Controllability in AI Models

What challenges exist in achieving controllability in AI models?

Controllability remains a significant challenge, particularly in maintaining character consistency across models. The development of Flux Context aims to address these issues by allowing for more nuanced prompting with both text and images.

  • Controllability is essential for user satisfaction in AI models.
  • Flux Context addresses character consistency challenges.
  • Innovative prompting techniques can enhance model usability.
# 70:13

Model Evaluation and Development

How should AI models be evaluated for performance?

AI models should be evaluated through a combination of human assessments and automated evaluations, with an emphasis on continuous sampling to understand their strengths and weaknesses during development.

  • Combining human and automated evaluations provides a comprehensive performance picture.
  • Continuous sampling is vital for model improvement.
  • Developers should actively use the models they create to ensure relevance.
# 84:16

Fine-Tuning AI Models for Robotics

What is the process of fine-tuning AI models for specific tasks in robotics?

Fine-tuning involves adapting a general model to specific tasks, requiring significantly less data and time compared to training from scratch. The focus is on context learning, allowing robots to learn tasks efficiently from minimal demonstrations.

  • Fine-tuning can drastically reduce data and time requirements.
  • Context learning is key for efficient robotic training.
  • The goal is to enable robots to learn tasks with minimal input.

Transcript

0:00 Happy to bring to you our episode with Black Forest Labs. Our guest today are Andreas Blackman. He also goes by Andy who is co-inventor of latent diffusion the technique that eventually got scaled up by stable diffusion and stability AI and they eventually split with a bunch of the core stability AI image gen folks to form blackforce labs and Dustin Podell who I also met with an at one of the stories that we tell and who worked his way up from just being a regular engineer a freelance like web developer person into a model and research lead at stability AI and now blackforce Labs, creator of SDXL, Control Laura, post training lead on SD3, Flux Ultra, Flux 1, and Flux Context, and we go into all of these details. as you may know, sometimes we're very honored to have an extended history with some of our guests, and this is one of them. You might have heard in our Anjida episode where we talked about introducing each other at one of the neuropsises and meeting Black Forest Labs for the first time. And this is right after stable diffusion.

1:04 But what you may not have heard and we only tell this story in this episode is how latent space itself was started because of stable diffusion directly crediting the launch for me basically quitting my job to go all in on AI based on what we saw. And today's guest Dustin also had that experience and so did Vivu. And so we have a very very strong tie with the team that basically led up to Black Forest Labs and now we're also interested in what's next for them like they have created some of the best open and closed weights models for image generation and video generation. I think especially flux context if you're not aware of image editing models. I do think that this is in the in the limit one of the things to fully replace Canva and and Photoshop and a bunch of other workflows and editors that you might use in a normal creative workflow.

1:57 It is not going to replace the top end of the market. But for regular normal everyday edits, why would you not want anything else other than to query your model and ask it to do image editing for you? we do this obviously with this kind of video editing. It's all clanker done and I do review it and they're also moving into robotics. And so this ties in with a minieries of model labs that have been pursuing this direction building world models from image gen life with runway as well as video models and now robotics models and so I think a lot of things rhyme in a lot of our recent episodes like even the artificial analysis episode that we're going to release soon they're also going to robotics and like it seems pretty obvious that when you see a whole bunch of smart people who are at the frontier all leaning one direction that you're probably 6 to 12 months out from some kind of really big breakthrough. This has been a consistent theme over my last four years of podcasting. It's not going to change now, so look out for it.

2:56 that's the most obvious call out I can make, but there's also a lot of interesting lore for people interested in the general diffusion space. U, we're going to do a little bit more over time. I'm thinking about an AIE LA where we talk about the generative media field and the creative arts and how the complicated relationship is evolving and I think maturing to be honest. And finally, for those longtime listeners, we are starting to open up to advertisers, people that we really admire and respect and want to showcase what they're working on. the first one was with Superbase which I think a lot of you enjoyed and the second one today is brought to you by Dgram and you'll hear more from us at the midpoint of the episode. You should be showing a robot in action one time and then it should be able to do this and this would be truly a superhuman intelligence like at stability that we couldn't essentially leverage the value of what the models could be as well. Mistakes of not being able to actually capture the value of the models themselves. We raised all in all $450 million in funding so far. Currently the last that was published in December was more 100 million revenue. PFL was not founded to be an image generation lab. Video modeling is actually the central means to understanding how our world basically works. Maybe like 6 hours of training, 4 hours of train. It was it was very very short training run. And all of a sudden, like I saw what felt like a clear generality coming out of it because it was >> Okay, finally our BFL pod. Welcome guys.

4:28 Andreas and Dustin. >> Great to be here. Thank you so much. >> Yeah, definitely. Thank you for having us. I know we've chatted about doing this for a long time. >> Yes. So, we've had an on a couple times and that's where I met you at Vienna. I think one of the ICMLs or >> was this like one of the like was it was the like rooftop event or >> there was a year where there were two Viennas back toback.

4:49 >> Yeah. Yeah. Yeah. >> That was first eye clear and then it was ICML. >> I don't know how they do this thing. >> Anyway anyway and at the time you guys had were like just about to launch BFL like we had a dinner and like you were like oh we're thinking about what's next after stability and all that. So anyway congrats. How do you like guys like to introduce yourself? Maybe we'll sort of get a crisp self intro up front. So I'm Andy. I'm a co-founder of Blackboard Labs. Been developing diffusion models for oh wow I think seven years now. U met Robin and Patrick way way back in the day. I think it's yeah 7 years now since we met at H Highleberg. I'm from Germany originally and there we started looking into yeah just diffusion modeling for images.

5:32 we're all having a kind of a physics background in some sense and this this paradigm of just like mapping this general physical phenomena of diffusion to machine learning and generative modeling had a a kind of a philosophical kind of beauty to us that's why we got into this and then yeah developed a lot of the I guess fundamental algorithms that are powering diffusion models today or image generation video generation and also action prediction today I guess we'll talk about this later.

6:02 >> We'll talk about it. Yeah, of a kind related to it's not immediately obvious to people image generation to action prediction. >> Yeah, >> but we can talk about that. and then one thing I did want to comment is a lot of people don't know the relationship. Black Forest is a forest in Germany. It's a mountain range actually, >> but yeah, it's it's a region in Germany basically named after and funnily Robin and me we we first met during our PhD and then we figured out we were actually born I don't know 10 miles away from each other and that was that was pretty fun and then eventually that led us also to come back to Germany.

6:37 >> Beautiful. >> All of Europe looks like this. >> You can see the lab right in the middle there. >> Yeah. >> Yeah. Yeah. >> so you named it because you're from here. Exactly. We named it because actually, we founded this research lab and back in the day we could basically pick where we wanted to do it. No one said us, okay, you have to come to say SF or whatever place. So we decided, okay, we want to be actually starting this at the place where we were born in Fryborg, which is like at the foot of the Black Forest basically or a city in the Black Forest and that's how it came and that's how then the company also started there.

7:13 >> Yeah. Confis also H highleberg >> confis was in H highidleberg exactly that's where we met we then later moved on to Munich that was all >> which is the team that did lat diffusion >> through our PhD exactly and and that is when we developed this latent diffusion algorithm >> I really like your name by the way your podcast name lat space I I I I would actually be curious how that came together >> I can talk I can talk about it but yeah go ahead >> and exactly we developed then this lat division algorithm at the university and then based on this research project we scaled bought the algorithm which led to stable diffusion then effectively and yeah since then it's been a constant journey on a rocket ship you could say which is super super fun to to have and we've met amazing people like Dustin as well >> who's who's one of the the kind of oldest and and like yeah most I think relevant researchers that we have but yeah we'll we'll learn more about that >> we'll guess him up later I think latent diffusion and just generally like VAE as well and and then for us Lin space is this general idea even our logo is actually a VAE and it's this idea that you can translate things into spaces and then explore things that are adjacent to what exists today and then you can sort of re translate back >> and I thought that it was like such a cool insight for creativity and generation that I never really understood until I studied stable diffusion like the launch of stable diffusion changed my life I started in space because of stable diffusion I quit my job because of stable diffusion Oh wow.

8:42 >> Cuz I was like, okay, everything I know about AI is wrong because I can run this on my laptop >> and like if I was wrong about this, what else am I wrong about? >> so there there's that. but then there there's another relationship with Terry Patchet which I I don't know if people have >> writer he has this concept of Lspace which is library space u where he has orangutan that's like the librarian in unseen university which is wizard university and it basically is a library with all possible books. So if you have a library of all possible books, you can just theoretically go in there and find a book that is right for you. and that is lat space.

9:17 >> so I originally saw called Lspace. one of my readers donated late in space and then became late in space. >> Amazing. >> Dustin like yeah what was your journey into diffusion and AI in general? >> Yeah. Yeah. So I I came from a little bit of a different background than than Andy and Robin and team. So I'm I'm not from the German side, but I've still had a fantastic time working with these guys. And honestly, it's funny you mentioned that you quit your job when stable diffusion came out because the original stable diffusion I was not working with these guys at and it was actually a cause for me to quit my job at the time. So I you caused mass unemployment.

9:50 >> I'm actually curious like the numbers of people that just like changed their life direction when stable diffusion dropped. I I would actually be very curious to know kind of the branching effects from that. But I don't know. We can we can theorize on it. That's actually super interesting because here in SF in the Bay it had this huge impact. Where is it? In Germany where we stayed back in the day that day, >> people didn't really notice it because like the people were so so far away from that. Like there are a couple of people obviously that that have been involved in the open source community also in Germany >> but the majority of people didn't really notice that drop in summer 2022 which was really interesting and then you see what happens here like yeah that that was to us that was really kind of super interesting.

10:35 >> Yeah. But but anyway, so I I guess to give a little background on me, so I I at BFL, I I'm one of the founding researchers that joined in with the we had a bit of a larger team that I'm sure we'll get into the history of kind of how that all came to be. But my background is not actually traditionally in, you know, the ML space whatsoever. I was traditionally a software research or a software developer for, you know, around 10 years or so. Early in 20, you know, around the 2020s, I became very very interested in just like open source AI. I saw like the kind of the early work in GANs and other you know in generative media and I was tracking with it. I was experimenting with it a little on the side. It got me quite excited. I mean and I remember following originally the the late infusion work. I was actually I was I was following that and quite excited about it. And then the one you know one of the things that kind of tipped me off to go okay maybe this is something a little bit you know a little bit more serious is actually shout out to Midjourney V1.

11:31 >> Wow. back when that came out and I I remember thinking like okay this this is actually going in in a direction I I can see this being quite impactful and then the one big moment the big drop that really changed everything for me as we talked about was the stable diffusion release and at the time I was working at a software studio down in Los Angeles stable diffusion came out and basically that day I turn in my resignation and said okay >> what was what was the plan like >> the the plan so so the way the way that I saw the way that I like envisioned it at the time I saw this thing that felt like it literally felt like a magical thing that came out and I viewed okay the people that are working on this must be like arcane wizards and I need to do whatever I can to understand like what magic or what runes or what things that they did to imbue in this to make this happen and let this go out to the world and have this have this impact. It wasn't like I was just totally disconnected at the time. I was in the stable diffusion discord. I actually had friends in there. I was I was kind of interacting with the community. So when was it viewed as basically open source majourney or like like do people not make >> there was I mean I I was in both I remember there were a lot of kind of discussions around this like oh stable diffusion was it's having such a you know there there was a lot of discussion and I remember the midjourney discord and the stable diffusion discord around that time around the the impacts it was having I mean it's been a long time so I can't remember the exact >> nature of it I don't want to speak on you know journey is doing fine like they're doing I mean they're doing great but for me it was more just like I was very interested in the fundamental technology and then the fact that I could now run this at home and then really the thing I saw was then just this community of developers actually like taking it home developing on it. I was hacking away at it. I was meeting a bunch of other really really smart great people some of which are actually at BFL now that have have come along for the journey. That moment of just community interaction and and learning and developing and building together was really an impactful moment for me. Lucky luckily enough for me I was able to find a kind of a job opportunity to get myself into stability at the time where you know that this this was starting and it it I didn't actually start working with these guys right away. It took me a while to kind of to kind of work my way up to that and we can go into that history later if we want.

13:39 >> Stability had a bunch of different teams right LM stable music >> harmony the audio team. It was it was like I don't know the way I would kind of describe it is like if you grabbed a bunch of Discord was kind of the space that a lot of this happened at the time. I don't know. I don't know why the whole >> It's like if a discord had it was a physical manifestation. It was a melting pot of very different people that came together. We're just like interested in this >> emerging new technology that came obviously and that's also how we like this is one of the most exciting things about our team, our founding team is >> it comes from very different backgrounds. like we have this hardcore academia research branch and then we have people who were from the open source community like like Dustin who actually got why a very different kind of trajectory into this space >> but I think this is what makes our team so strong. We have these diverse backgrounds and there's yeah there's rarely a team I think that that has this history also this kind of joint history together.

14:38 >> Yeah. Yeah. >> This is just amazing that that like >> Yeah. And we can go into more of your other co-founders first. I I did one. So I was also in that space, right? Like literally, you know, I was also a very influence of stable diffusion. I did the wrong thing. >> I basically went down the path of comfy UI where I build UIs for stable diffusion. You seem to have done the right thing where you went to research.

14:59 >> Well, it's funny you mentioned Comfy cuz like I used to work directly with I mean comfy join had the pod. Yeah. I think he he moved here. >> Yeah. Yeah. He he's like my neighbor. >> How does a He's a researcher. How do you become a researcher? I'm someone that's like when I get interested in something I am obsessive to the point of I cannot put it down until I pull it apart and figure out every aspect of it. And so to me I'd already been experimenting with pulling the models apart talking to a lot of different people in that open source space. So not just like developers but other researchers. I mean remember talk about another discord the you know the Luther discord back then. At that time I met a lot of people who I were I was just learning from and I was reading a lot of papers and just trying to uptake and understand for myself. Joining stability was also another major point for me. Originally I didn't join as a researcher. I joined as an engineer. But all of my free time I spent with essentially any researchers I was able to make friends with. You know shout out to Dr. Scott Holly there who worked on the audio team who mentored me quite a lot there. And at some point I was able to to sneak my way onto the onto the cluster there and got myself access to you know some idle compute. And I I I really just like I treated it like I treated how I learned most things which is just open source hacking trying things reading papers trying to implement different pieces of work and just seeing how I don't know how models trained how they learned and and I mean that was really it like there was there was no magic to it other than just I was excited and I just wanted to learn about it. I wanted to tear it apart until I understood how it work.

16:27 >> I mean this is open source at its best. >> Yeah. I think that that's generally how you learn best just trying out things and he was also >> Luther. >> Yeah. Yeah. affiliated but not on the image gen side. But back then everything was like very hackable, right? The the delta between open models was like >> pretty drastic between what was closed and good, right? So people on language model side were doing post training. That's where like a lot of the news guys started in this. You know, you can look at early stable diffusion.

16:55 >> This looks so bad and >> where we get to today. It's pretty crazy. But you both had pretty, you know, impactful roles. like you did SDXL, you did latent diffusion. We'll get through all the history, but >> for people that haven't super followed the space, where are we today? Like what's the what's the state of things? I know Flux 3 video is out. Flux 3 Action, we're now like 7B doing robotics. What's the latest in the space with you guys? We started as an image generation lab when we started Black Forest Labs, right? As a kind of legacy of stable diffusion and the likes.

17:28 and that has been I think our our thing for the first year but then we realized okay you actually image generation is nice but this technology can unlock so much more. >> Mhm. And that's when we slowly went into what we call visual intelligence, which is basically trying to get our intuitive intelligence that we people have when we say when we catch a ball and just understand physics without actually knowing how to mathematically describe it or something or when we grab a bottle and just know exactly how to do this because we've done it 100,000 maybe not 100,000 times, I don't know, but a lot of times in in our childhood, right? We want to be transforming that knowledge and that form of intelligence which is very different from language model intelligence into models and learning based on pixels which means not only images but also videos and maybe also audio say not pixels but natural signals is maybe a better term >> that really unlocks this and that is what we realized and we realized hey we have the algorithm to actually do this which is diffusion or flow matching and that is when we just said okay we need to extend and from image generation to basically all the natural representations which led to flux 3 basically which is one model that learns multimodal from images videos audios also actions we can talk about how this comes together later >> to really get this to to keep it with Daniel Canaman I don't know if you read thinking fast and slow it's a kind system one intelligence it's not system two intelligence that you could maybe >> system one models now >> may maybe language modeling is system two >> but we focused for now on system one which is extremely important for everything that is intuitive and that requires real world interaction. That's why at on a high level it's also so good for robotics. So that's been the journey and now we're really focusing on that and in the in in the end obviously it's about combining both of these systems into this broad very very broad way of intelligence that we humans also have.

19:29 >> I want to go into sort of history and all these things. but just to also recap for the for people at the front, you're already making a significant money as a generative media company, right? I don't know what the the numbers are and and maybe you can sort of recap there. You >> raised you raised a ridiculous amount of money. We raised I think all all in all $450 million in funding so far. >> yeah, so there's way more way more than stability. any revenue milestones that you've that you've announced? kind of the lowest currently the last that was published in December was 100 million in revenue crossed that since I can can >> clearly clearly have crossed that if you raise 450 exactly you know yeah anyway I'm just saying like you did not have to pivot >> and to be clear we're not pivoting we're just extending like this this model is really it's trained on everything >> there's always been the plan >> and obviously to to create artworks you also need this visual kind of intelligence to create a video that we see here >> you need to know that how physics play out. I I need to know that the model needs to understand this is actually interesting. but but the model needs to understand that the arm of this say this this this capibara or what it is is not just falling away that it has to follow some law that is described by physics. It's just not describing the law itself. It's just describing the output in a similar way that we humans don't understand physics but we understand that if I let a ball fall, it just falls down. Right. Yeah. Well, I think there's there's different concepts of a world model and obviously that's something that we can get into where can you learn this is sort of the bitter lesson version where you just learn on a bunch of input sources or you have to actually physically structure the 3D model and and kind of learn from that >> no no 3D model at all just learning on based on natural data. We we also we have a bit of triangulation like from our eyes right we have two eyes and that gives us a kind of slight understanding of 3D perception but effectively to a very large extent even if you would like close one eye we could still kind of learn a lot about the world that is what these models are basically learning like you just learn from natural data 2D video data that obviously has a lot of 3D understanding because if you want to generate a pan around an object you need to be hallucinating how it looks from behind and if I look at I have a pretty good understanding of how your back looks basically.

21:49 >> Yeah, >> because I've just seen it before, right? So, this is gives you an implicit 3D understanding, >> but it's really trained on kind of 2D videos. >> Yeah. I'm trying to steal the counter argument, the the opposite argument, right? Like this like the Gary Marcus like I'll don't learn anything or I I forget who are the famous critics of this kind of stuff, but basically it's like it's like I have never lived I as a model I only see on inputs. I've never I don't have personal experience. you can model my back because you are also a human but and you can model other humans but and maybe maybe this analogy isn't isn't exactly correct because I was obviously I'm sympathetic to your your side of the story but like I I think that is how people criticize this approach to world modeling which is that yes you can approximate closer and closer closer but you'll never quite get something core to the to the existence of the thing because you never modeled the thing you modeled the most likely you know sort of constraint around the thing.

22:45 >> Yeah. I mean, I I think to some degree you could argue that we operate under the same principle, right? Like you can say like that we are, you know, okay, we have this embodiment. We're constantly like taking in data and like obviously we we regularly update, but in some principle like we're still just taking in these like filtered signals from the world. I mean, there's all sorts of wacky things with how our our eyes process and we have to, you know, part of our brain kind of has to hallucinate some of the info because we have the blind spot in the center and and and other aspects of this. But fundamentally like I don't think there's that much of a difference between you know how we like we're we're out here taking in this natural data and building this model of the world that we're we're able to act on versus like these models also learning from. Obviously, it's the signals are different that they're, you know, okay, they're turned into to into pixels or, you know, into into waveforms and process through this other mean, but like the the information is still there for the model to take in and build this like representation of the world that then you can essentially operate with.

23:39 >> And yeah, I was actually arguing this with Lucas Bayer. he's also he's with Meta now, I think, formerly Google and OpenAI. He's in Zurich, which is pretty close to Fireborg, and he he's coming over from time to time. We were we were debating this on on human learning like he was he was we were saying okay we actually don't need interaction which would be kind of the I think what I would say is important for learning like higher forms of intelligence it's like first it's observation that's also how we humans learn when we're babies right we just lie around first and we we observe things and from that alone we learn so much this is what what you can actually exactly map to what how we're currently training these models they just observe a lot of 2D 2D videos, audio and then they learn from it. And I think next step would also be getting closing the loop to actually interaction with the world because that's also in how I think that that's very important in how how we learn basically around the world. We can then tap and we see oh there's a correlation between >> I knock on wood and there's a sound attached to it, right? I I can just like manipulate the world then and to to understand it better. That's how I think how humans are mainly learning at least how I personally learned. and he was actually saying he thinks the majority is observation which I found very interesting because that would that would if if that would be true then we just need to do what we're currently doing more and that could basically unlock an insane >> amount of this is like emergence with good priors right we've seen adding new modalities to say language models you don't get like the full benefit of you know doubling your performance since you've added vision to a language model but if you have good priors you know is there light benefit. Is there a big benefit? It's it's interesting to see. I think there's a lot of technical stuff that'll be interesting to walk through.

25:25 Like gshian splatting was a big thing. World Labs did a lot of 3D data. Where are the trade-offs between video generation output versus what else? But I think something fun for listeners of the podcast would be the history. You worked on, you know, very early stable diffusion, SDXL, latent diffusion even before that, >> video diffusion as well. >> Video diffusion. Basically, there's a lot of buzzwords here. you know LCM Laura flow matching old prompting can you teach us everything no what's interesting like how big are these scaling laws >> so I've actually built like a tech tree like I wanted to build this game where it's like civilization but the tech tree is AI like key AI papers so to spend research points and get it and so like these are the tech trees right like first it's GANs then you know dot dot dot like the indiffusion happened but you like there's a lot of core points beyond that.

26:20 >> I'd play that game. >> So, so, so, okay. Anyway, just to pitch the game, it it's like you role play as a Neolab and you have to you have to spend on like GPUs or researchers and then you have to collect data and like there's there's all these like little things that you can research. Yeah. >> I come on, whip it up. Let's we're in the V coding world. I want to play it tomorrow. >> It turn I think game balance is really hard because I had you guys played I I have I I've built it.

26:45 >> Oh, fantastic. But there's like game balance is really hard. It's actually like a AGI hard problem I feel like because you know once you find a way to win they just always win like that and it's not interesting as a game. You need multiple ways to win and like you need high-risk ways to you probably lose but maybe you win. >> Yeah. Maps classic problem in dance by the way. So I think I think one thing that VU is interested in I'm interested in teaching our audience like like what are the key points that like if you're if you've heard a lot of terms like what are the key things that have really impacted the field and you know a lot of you have you've lived through a lot of >> I'd also say this is a great way for people that want to get into the space good interview questions if you're hiring as well like you should know most of these things I think there's some points where there's like spicy you know different paths that's popped up so auto reggressive image gen versus diffusion two backbones or glassions all these things. What are the key things people should know? You know, I'll leave it I'll leave it pretty blank.

27:41 >> I mean, first would be SDXL maybe talking about SDXL a bit. I >> was going to say what were the what were the things we did in STXL? >> Oh, no. Please not the refiner. >> Wait, why that reaction? >> So, SDXL was a big piece of work in trying to make the models feel good to use for the first time if I was to kind of like lay anything out. Not to say like stable diffusion one two weren't you know I mean they were very exciting they were very fun to use but to anyone that was like in the space at the time between the period of SD you know SD1 to SDXL we saw a ton of work out in the in the open source community working on how do we fine-tune these models how do we make these models you know more aesthetic how do we make these models into like you know multi- you know if any remember is like the original stable diffusions were only able to do square images and so SDXL was the first work of trying to actually release like a found like the base foundation model kind of with a lot of these general improvements implicit in the model. So SDXL involved you know the first time we did large scale multi-aspect training there was a you know at the time this was before packing there was like an efficient way we could do essentially bucket style training so training you know on all sorts of different resolutions of of images in you know a large scale this was the first time we did multiple actual training stages so taking it from pre-training actually developing like post-training sets so the there was a lot of data work that was done in SDXL all I'm trying to think like all the other aspects and then I'll I'll give you kind of like a little bit of a backstory on the refiner and maybe maybe some things that didn't quite make it in. So here here was the whole process of the final release model is essentially SDXL fundamentally worked similar to the original stable diffusions. It was a latent diffusion model took in text. It had you know you had a noisy latent took the context of the text actually generated the you know generated the imagery in latent space and then a VA decoded it and then from there the refiner was actually another full diffusion model that the the whole point of it was to take essentially what was not a rough image but SDXL wasn't the best at what we would call high frequency detail which is this like the fine detail like skin texture texture of grass upscaling. It wasn't an upscaler cuz it didn't actually change the resolution, but it it >> yeah, upscaled in terms of in terms it refined it.

30:00 >> The reason I thought it was janky is cuz I worked I not just me the whole team that worked on this was was fantastic. I don't want to take all the credit at all cuz there were there were a lot of fantastic people that that worked on this. We did a lot of work to try to push the detail quality of the model while trying to keep this like generality and we just didn't quite get there. And so then there was this like side track of work of like okay what are other methods to do this and this is where the refiner came into being with actually the refiner was actually a model that was it was what was supposed to so this is a little bit of the secret lore is there was a version of SDXL that was never released it was a different architecture that didn't quite work as well but it was trained pretty far along even to like the post- training stages the refiner was that model it is a full diffusion model >> you completely forgot about that >> yeah it like the refiner was a full diffusion model and the one thing about that model is it had a little bit of better performance on these high frequency details. It just had very poor structure. And so this whole idea came of like what if we essentially like >> different model, different job, >> different model, different job. We we took it, we did a lot of work to really push the, you know, the work on how well it could actually refine this high frequency detail. And then we made this this essentially this pipeline. Yeah.

31:12 Which is the XL part >> which which then became okay this together is like a more performative model, you know, or a more performative system altogether. And then that was what we ended up releasing here, >> which is actually interesting. It was one of the early pioneering works of like actually that looked into okay what kind of properties learned in which part of the trajectory of this diffusion kind of trajectory right because you have these structural things that are learned >> closer to the noise if if you come closer to the noise and then you have the fine high resolution high frequency stuff that comes further down and and is closer to kind of the clean image basically and like today there's a bunch of different books that actually analyze this but you do actually you you kind you generate from low frequency to high frequency if you go from like noise to clean image right along this trajectory but that was not really known like it was informally maybe a bit known by then and this was it was one of the first works to basically exploit this and say okay yeah I actually I dissect this kind of trajectory and I I have two kind of experts one for the structure and the next for the texture basically >> and yeah right now we have modulation which is like condition on the time step to actually account for that as well and stuff like that in the architecture but that was really the first one of the first works to to really make this distinction and it as you see it worked quite well.

32:28 >> Yeah, it did work quite well. I guess the reason I I still hold frustration because at the time I wanted this one model to be so good at everything and I felt >> Yeah, maybe it was like a little bit of a copout. Now nowadays I take a a bit of a different view cuz I mean we see systems and models do incredible work together but at the time I remember feeling and I still just have this like kind of old frustration you know just just from back then it's also very fair because you should always go with the simplest solution that works and if if you then have a kind of a you need >> at which point do I need a machine refiner. Yeah, this is like when it comes to scaling, whatever the simplest >> solution is that works, you should go with kind of a bit of a razor as well in in statistics basically.

33:12 >> So, you know, runway was also kind of doing their gen one and gen two and Anastasis on on this part also said admitted that gen two was pipeline like they just didn't admit it whereas yours was public so you had to admit it. I think if you go back to the time too like people I don't people coming into the space today haven't seen 200 500 steps of diffusion right like as you say like you have a background and then minutes later you have little fine details coming in it was also very tinkerable right this came out as what like a 3B works on 8 gigs of RAM so as much as you could bit or less and do the general approach if you're not a lab and you're just doing this at home you you have like one or two GPUs you're going to do pipeline Right. It's like that's how you can see the differences right there.

34:00 >> Yeah. Yeah. so video I've been working on video for quite a actually already before stable diffusion. I was working on video generation for Nvidia. had a paper there align your latence where we basically map the was actually in parallel to stable diffusion. We basically mapped this latent diffusion algorithm to video which was the the natural next thing to to to do. It was very exciting to kind of then just at once keep things moving and you can imagine the kind of compression reasoning behind why stable diffusion was so efficient compared to other models in the in the time also holds for video where you actually have this temporal dimension that you can also compress right so that made a ton of sense. worked on this for first for Nvidia and then when we when we basically joined stability we basically after I think after SCXL >> Mhm. We worked on this this stable video diffusion which basically first applied this this latent diffusion approach to video asset. But importantly, this this work was a lot about investigating the effects of data filtering which as we felt or as I personally felt back in the day were like in research and they still are very much underappreciated like data is just like when we think about what these models are doing effectively we're estimating a probability distribution and the data set is what gives rise to this probability distribution right and if you have the perfect algorithm the best algorithm in the world that that just learns anything you But your data is very bad and you have like on average blurry videos or images and like like not a lot of motion in the videos, your model will learn that no matter how good your algorithm is. So what comes out of the model will actually be very bad and like like not nice. So the data is one of the most important levers that you have as one who who wants to train these models. And in the entire research field, there was never much talked about data because a lot of people from the research community, they just worked on benchmarks and like tweaked the algorithms. But if you talk about frontier models, obviously we know that by now that data is extremely important, right? And and that is what we wanted to be really like sharing a bit like sharing more context on. So we investigated how much data like and then you have this this trade-off I want to be training on a lot of data >> quantity versus quality >> exactly where is where's the sweet spot that was what we investigated there and then also a very interesting chart >> we should pull up >> coming coming we can actually take it the most exciting experiment that we did it was not the data one but actually if you go down a bit >> we then and it comes back to what we discussed earlier we then showed that just pre-training on plain 2D videos allows you to fine-tune the explicit 3D model extremely fast from it. So if you go to the to the last paragraph here of the of the paper, here we go. These are these 3D 3D views here of of the model. So what we actually did is we took this video model that we trained and then we had I think 1k fine-tuning steps on 3D data and we got a 3D model out of it. And and that that comes back to the question, okay, what what do you actually need to to kind of understand the world in 3D, right? Only one K finding that shows that this video model although only trained on on plain video actually had this insane amount of 3D understanding and you just took a bit of a a focus to get a 3D model out of it and that that is really like that was one of the early experiments where we saw okay this is actually kind of you can get world understanding through training on plane video and like from a research perspective this was like I think the most interesting experiment in this paper >> also pretty short paper there's like a section here where you even explain the pipeline, right? There's like three, four stages. Pre-training, some post- trainining, yeah, stage one, image pre-training, video, post video pre-training, >> video fine-tuning, and then 1,000 steps, you can also make it 3D, right? Crazy.

37:55 >> at the time, I I don't remember what I did, but there was basically this temporal consistency problem where if you just stretch out diffusion and try to stitch frames, it wouldn't work. So a very old hackathon project where I I forget the method we even used but we fixed this temporal consistency to like scale it out to 15 20 seconds. It took forever but it was a very hacky solution cuz you could do this in an afternoon right. But you critically you need this kind of >> learning over spatial temporal kind of over the temporal dimension. Then if you just stitch together kind of a very fast diffusion model you don't you have this flickering even even if you you fix the seed and you do a ton of effects but you realize you need to like in in currently every or today everyone is using transformers so in the transformer speak you need the attention over the temporal dimension as well to kind of learn from that right and then understand what is actually >> what how does consistent motion look like basically >> and as far as I understand we asked this of the on the gro imagine part as well there's no compression in the video side that is useful right you have frame frame frame frame frame.

38:56 >> Exactly. >> It's the simplest for version but it's >> I mean well well if you then look at MP4 codecs you you have very smart compression but that is lossy obviously. Yeah. and and you need to kind of you need to connect the frames in a way and like like extract what what what how how how >> how kind of consistent motion looks like basically I don't think any video model exposes this but kind of how far you look back in the temporal attention because sometimes you do need longer or it needs to be more adaptive maybe for example if something gets oluded and then comes out again you want you want them to be the same but what if like how long is it oluded >> I mean principally that that depends on on the on problem you want to be solving, right? If if you if you want to be creating a kind of a a long memory model that that that can can create your virtual worlds, then you obviously need to find a way to look back very very very long.

39:46 >> Super memory. >> That is that is just I think also one of the most exciting research problems to work on these days. Long context is just extremely important. >> I don't know if there's more there before we cut over to Black Forest Labs. anything else in the stable world? >> S I mean there's S3 that was the last paper that you did before you >> Yeah. Yeah. SD3 was a I felt like a pretty big paper >> that that was a was a I think one culmination of our work of man stability I would say like >> yeah absolutely >> we your graduation thesis >> if you want yes I mean >> no this this paper was really about understanding more in which part of this trajectory from data to noise does the model actually learn what properties and where do we have to spend most of our computation that we put in training >> to maximize the performance of the model.

40:32 >> Yeah. So before and obviously that was also the transition from classic diffusion to flow matching with a slightly different paradigm basically >> you call it rectified flow. >> Exactly. >> this was also the first time I used like diffusion transformers right with the a bit further. Yeah. So lot lots of changes like going from a convolutional architecture to a transformer architecture changing the probabilistic model from from a diffusion to basically flow matching which is way simpler mathematically which is super super nice >> I think you should actually define some of those terms for people that haven't heard latusion you know flow matching what how did you come up with the concept of these ter things too right >> well he didn't come up with flow match we we exactly we we didn't come up with flow matching we just saw this algorithm as something that is in a way more general because you can like for diffusion you always need one of your two distributions between which you're basically translating needs to be a standard normal that does not hold for flow matching anymore and instead of approaching this kind of problem to to to transform one probability distribution into the other with a seeing it as as a solution of a stoastic differential equation. you're actually seeing it as a a solution of an ordinary differential equation which simplifies everything completely like mathematically completely and makes also fast sampling much more feasible you could say because what we always do obviously is when we have trained the model we want to get be making as many steps as little steps as possible to basically sample as fast as possible right some people would say it's it's actually a special case of of diffusion flow matching other others might say okay it's actually a generalization of flow matching of diffusion the flow matching algorithm because you you you go from you can translate between two distributions no matter how they are shaped and you don't need this this kind of standard normal basically guess to to be one one of them but yeah in general it can be seen as again a a model that learns basically to a vector field between two that that that translates between two distributions and once the model has learned that vector field you can just integrate it numerically which is then as easy as solving a OD by means of an oil or forward step basically. So it's it's it's very similar in in many respects.

42:52 but it was just like I would say mathematically easier more general from from the the the kinds of probability distributions that it can can translate between and that's why we basically went with it and maybe as a third party said it's also easier to derive a fast model from those flow matching models. Basically, >> it was it was much simpler and more scalable like it was it was it was much much easier to work with and then had this like nice performance benefit as well. So like I don't know there the the the diffusion space got quite complicated.

43:23 >> Are we satisfied that there's no loss? Is like is it is it lossy in acceptable ways? >> Lossy you mean lossy compared to full diffusion or >> Yeah. Okay. So then it's good. It's a strict improvement. >> Exactly. It's it's it is an improvement. >> Yeah. I mean it's it's effectively just drawing a line between like you know noise to data really. It's it's it's like it it it kind of took this like there was this complex space on like how do you map between these two the you know the like these two distributions as Andy was saying and it simplified it down to like the simplest of just drawing a line >> because you're trying to do a shortcut like what if it's a bumpy distribution.

43:58 >> Yeah. Yeah. That's that's the thing. >> And like the most important part of the paper or one of the most important like on on the on the probabilistic modeling side was actually looking at the trajectory from noise to image and which parts really matter and which don't because if you just like we we we train this in kind of steps, right? We we we always on on this part from data to noise, we only always train on one kind of step and then we approximate this direction in flow matching with a model. And if you just sample those steps in between data and noise uniformly, you might actually spending a lot of time optimizing for things that actually don't matter. So one one central part of the paper was >> where should we spend our compute? Which parts of this trajectory from data to noise carry the information that's most important >> because there this is where you should spend the bulk of your compute at training, right? And over sampling these parts during training and then under sampling other parts that are less important. It's it's effectively like a further like a a much further better exploration of part of what we talked about with SDXL which was like the refiner we spent time like training the refiner on the end of this trajectory to learn these high frequency details and this was like more of an exploration of the entire space of where like you should spend time to make the most like general good model to learn the data.

45:14 Yeah. >> Amazing. one thing I did pick up with SD3 was text in images which you guys showed off a lot. There was a time when images could just straight up not do text and then they did more and more text and then now it's like you know here's here's here's our nice typography. I feel like it was can we attribute that to the diffusion transformer or >> I I don't know if I could say it was the diffusion transformer. I mean I I think it really was just >> better model. It was better modeling and better data like it was fundamentally like as much as much as I want to like as as much as like I don't know you always want to attribute some fancy thing it really was just that it it could model the world better.

45:50 >> It's a data thing. It's also like what you always need like for everything that that you train on, you need a caption, right? If your caption doesn't describe the text that is visualized somewhere, your model will not learn it, right? So, that's also something that that is often >> sometimes we spend a lot of time talking about the nitty gritted details of some algorithm. In the end, it's again the better lesson, right? >> Simple scalable is is almost always the most effective. I would say always the most effective.

46:16 >> Yeah. But you can obviously scale in a smart way and still if if you if you spend tons of compute you should be spending it as efficiently as possible and only where it matters and that is also at the heart of this this paper basically right >> I think prompting also got a lot easier at this time we used to have the >> 4K yes >> who was the Greg guy the poor artist >> and now you should >> Greg Rowski yes I wonder how he's I hope it's doing well. I hope it's doing well.

46:47 >> Yeah. >> Yeah. >> And also, you know, for context, this is when Dolly 3 midjourney ideoggram v1 nothing was open. >> So, anything that you would see of this quality was either old stable diffusion or >> pretty much a closed closed closed tool. Yeah. >> Then you had to make the big break to Black Forest. what was that whole period like? Right. Like basically you had to leave and start a lab from scratch. >> I think it was probably one of the most exciting times. we all had because when you're in stealth, you're just like you're a small team. We were >> 13 13 people. I think when when when we started, we first went to France, had a had a had a a fun little week in in in the mountains in France in a in in a chalet.

47:30 >> Spent the entire week just hacking, hanging out. Dustin figured out some really cool stuff to make our models way more consistent. I I I I remember that. That was so fun. That was a magical week. That really was our kind of founding off site. That is when we started training flux one which 3 months later resulted in this nice company launch basically with a model that that I think blew pretty much everyone away. at least it blew us us also away. Oh yeah.

47:58 >> Like commercially what what this directly unlocked like only only having having kind of incorporated the company for 3 months and then releasing this model and it was like yeah just like was received very very well. Right. That was that was amazing. so that was >> but like did you know that when you started this I mean you had to get all the the you know the clusters together like all the data you had to start from scratch scratch.

48:21 >> We had to start from scratch. Yes. But we knew what to do basically. >> Yeah obviously >> we we had a lot of experience. and >> but like do you did you have a you just did SD3. Did you have a hypothesis like that you were exploring >> of like what you would do differently? We we we've obviously gathered a lot of lot of knowledge like in these entire times even even before stability as well, right? And that all culminated and then we also had a team that that just like was in one place basically that was very focused which had never been the case before because like that's just like if if you're working from from very different parts of one company also as Dustin said Dustin was actually not in our team before >> and that like we basically had had a super nice team together spend a lot of time together discussing how we would actually design the the the kind of learning process of this model which was at this off site and then we just kicked off and that result.

49:14 >> Was it like discussions of what is flux one or was it broader where do things go in a few years? How much of this vision was like >> well I I guess if I can jump in and say like I mean one of the reasons the company was started I I I will say there were some frustrations of time at stability. You know we saw from our our side we saw the impact that our models had made out in the world. We saw not just the impact from like the open source world but also we saw companies that were started and run and and and quite successfully used our models out in the world like we knew there was a lot of draw and a lot of potential in what our models could do and unfortunately I think we also felt that at stability that we couldn't essentially leverage the value of what the models could be as well. So, not only did we have the belief in like our team's ability to actually do the research and perform and build these models and and get them out into the world, but this would be our shot to kind of correct those inefficiencies or mistakes of not being able to actually capture the value of the models themselves. And and so that was also a big part of the belief of why we thought we could, you know, go forward cuz we I mean we had this record of doing doing this and making this impact and we figured that could continue on and we could actually make this you know successful company out of it which now I mean I think it's I think it's gone pretty good thesis.

50:33 >> I think it was a good thesis. >> As I've said before, >> yeah, >> we've been talking about how we can generate images and video and improve the perceptual understanding of models, but what about voice? And that's the subject of today's ad. we've been working with Deepgram Graham for a while for on the AIE side and they recently acquired Pipcat which has also been a big sponsor of EIE. So Dgram is becoming one of the critical pieces of voice infrastructure for all agents. So this episode is sponsored by Dgram. If you're building a voice agent, you needed to keep up with how people actually talk.

51:06 That means understanding words in context and keeping latency low enough for a conversation to flow. That's what Deepr speech models and real-time infrastructure are built for. One in 5,000 organizations uses speech to text, text to speech, and voice agent APIs in production. The focus is accuracy, low latency, and cost efficiency. So you can build and run voice experiences at scale. You can use the cloud APIs or deploy it yourself, including on premises. And there's an offer on Flux text to speech. Spend a dollar, get a dollar in credits up to $500. To sign up for free or find out more, go to dgr.com/context is everything. Dgram. Context is everything. Thanks to DGA for bringing us this episode. And now we'll go back to the interview.

51:44 >> Well, and like what use cases were you seeing, right? Because I think a lot of people went straight to like Hollywood like we you know that's our goal or like IMAD used to say we will make Game of Thrones season 8 again. >> I mean I I'm trying to remember this. I can I don't know if I'll be able to remember the name of this company. This this might have even before SDXL. but it was a company that for like a week or two it went viral where anyone could take like a picture of themselves and then it like created >> the older version or so. the older version and it used like the old dream booth style fine-tuning which was extremely inefficient and then but it it like blew up like everyone was using this everyone was excited about it the models could make so many things and we saw that there was a lot of potential and how everyone wanted to use it and you know we we saw where the models were going to you know at the time they couldn't do typography we knew obviously okay typography is going to be important and we we eventually got there with SD3 and and then so on so it wasn't the just one use case it was we saw that this was like a general technology that had a lot of potential and that was going to continue to grow and have potential in the future which now is expanding into you know the the physical world a lot more and but yeah this this was actually already the potential we saw that like >> yeah yeah absolutely >> BFL was not found to be an image generation lab it was more of a kind of a matter of the the the flux model being so good that we then capitalized on it as well because it gave us a kickstart >> but when we pitched the company to investors for the seed series it was clear we want to be working on video because we see that as a I think we shared this on a podcast actually back in 2023 already when we we're still at stability. We saying okay this like video modeling is actually the central means to understanding how our world basically works because again what I said earlier right essentially we humans are also first observing the world then starting to interact with it and by that we we learn so much so that was always the thesis and then obviously we had to kind of take this detour which was very successful I would say to through image generation and eventually and then you always like face the the the kind of the issue of okay where do I spend my time on that that's unfortunately a thing we we we we cannot get across that the day only has 24 hours unfortunately >> but eventually I think flux 3 is really like where we saw this going for a long time and what you wanted to be doing for a long time >> and that's why we're so excited about like having the model out there being able to create videos being able to create images being able to create audio being able to act as a robot brain to actually kind of work in real manufacturing workflows and there's there's a lot of stuff coming this year still >> that we we have to announce on on on on the physical AI side but really like as Dustin said our goal was always to create the most general technology and we see this happening currently which is why we're so >> yeah so excited basically >> yeah I mean I think you guys also took the right paths to stay focused right like if I'm not mistaken flux one launched with XAI it was their image gen model good partnerships before they had their own but like you know that's a good way to solve do we build distribution ourselves even to the latest Lux 3 Max is postrained with foul they do a lot of infrance distillation they're good at that right so it's interesting but it seems like you're expanding now towards you have what's the latest one with robotics >> flux reaction action before we get there's one more one to cover which is image editing in context >> sure we have a K >> also really nice history right here on the website I was actually That is >> it's it's weird. We're now at the stage of company that like things just happen and I I'm like, "Oh, we have this."

55:16 but I was I was kind of sad flux one wasn't on there to view, but it's like the original one. But yeah, but it's it's fine. It's I think >> even just the visuals here, like maybe someone will clip it off original stable diffusion XL to all the you see the progression. But one thing I do want to cover at some point, I no longer personally see the progression. Right. You show me state-of-the-art image video models. A lot of Chinese labs are putting them out. It's hard to eval. I know people care. I've talked to Sumit at BFL about. Okay. No, actually people in the industry aggressively AB test industry. There's there's a lot that people care about on evals, but there's also a level of it all looks very realistic to me.

55:58 >> the rest is prompting skill issues. >> I think there's a Yeah, I think there's a lot of little nitpicks like there were hands there's still issues in tech sometimes. So, >> there still hands issues. >> No, there were hands issues. There there's issues in perception and layout like I'm redesigning my living room. It's horrible. >> Controllability is I think controllability is the the thing we need to solve like large scale. There's so many ways you can control these models and maybe that that is a good sec into into flux context as well because that that was a pain point that we saw for a lot of time. People were training so many Lauras for just to get this character consistency into models, right? I had to replicate back then with you know all the >> and then we had this amazing pitch of Dustin. It was also north side also in France actually.

56:42 >> It was it was another cabin where I like a lot of things happen in >> cast we should go to >> we really like >> you should do more. >> Yeah. >> And and then we had this pitch of Dustin who actually kind of saw this potential and okay I need to I need to get this like context. I I I need to be able to prompt not only with text but also with images for image editing and that you know.

57:01 >> Yeah. Yeah. No, I mean I so I did this like I I >> prehype as well. I've heard the story that flux context was like you and one or two other people. >> Yeah. So for for a long time it was basically my side project and then I stole one of our one of our data guys a little bit into it and said I'm come we're we're building this thing. So for me like so so yeah as Andy was saying it originally started as a pitch of mine at some some cabin in France that I did essentially like a I I like to do these presentations every you know every offsite or every time we get together where I try to look at like the state of the field and where I think the field is going and kind of where like what what is something we're not doing or something where these models have a potential to do something novel and useful that you know maybe we're not looking at right now or maybe isn't solved. And so I actually did a presentation on I know this is like the most overloaded word in the world at this point but I was like oh it's the omni the omni model presentation at that point because there was a lot of research work out there at the time on you know other editing models if we remember back to I think there was the old prompt a prompt and then there's why can I not remember the name of this the pitch was essentially these models are clearly learning like these deep relationships about the world and this is going to keep coming up it's about these these relationships and to me you know like the relationship of like okay this cocaine hand is sitting on top of a table. This bottle is sitting here. And you prompt the model and you go, okay, I want a photo of a dog wearing a hat and it needs to know, okay, the hat is on top of the dog. It it needs to have this general understanding. And my kind of theory in mind at the time was that since the model already has all of this like spatial knowledge, if we essentially build the right data and train the model like train an already pre-trained model the right way, we should be able to evoke essentially like general image editing with very very little training.

58:48 capturing these already like pre-built relationships of when things are are there, when they're not there, and how these like relationships, this these spatial relationships inside of an image work. And then essentially what it became is like a three to form, I don't remember the timeline of me here over in our SF office with the one data guy I stole building out essentially just early test training data sets. And and I remember the moment I saw that this was like clearly working quite well when we made this effectively tiny data set of just very like highquality labeled. It was essentially describing like simple relationships in an image and that was really it. like it wasn't a way that you would prompt a modern like edit instruct model. It was just literally describing these relationships. And I trained the model for like I mean it was maybe like six hours of training, four hours of training. It was it was very very short training run. And all of a sudden like I saw what felt like a clear generality coming out of it because it was I think it was like a photography data set. It was just like photos of stuff but even just from that I was seeing like I can put a logo on a shirt. I can put like I can do and it wasn't good. It was just like the model was clearly learning to attend between these two concepts and use these pre-trained relationships. And then I took that to our next off-site, showed results, and then it became this essentially more companywide project for about, you know, a couple months where we really put effort in, brought people in, you know, really improved essentially. It was mostly a data set project. Like I'll I'll say it was mostly a data set project.

60:18 >> And then in the endation, >> and then there was a lot of there was a lot of very nice distillation work by Axel that was done to to speed it up. Then we essentially were able to put out put this model out in the world. And I I was quite proud of. Yeah. It was the first kind of model to really do this character consistent editing right in in like high speed that made it again accessible to so many people.

60:38 >> For most people, it's very common to generate something, chat with it, change it. Before this, image generation took quite a while and you needed to recreate the image. You couldn't change the >> same character. You wouldn't be the same. You >> or you need to mask something that you don't like and then fill in. >> Yeah. I just I wanted to get away from the slot machine. Essentially, everyone had to do the slot machine prompt and prompt and prompt and prompt and it was very clear that like iterative editing was like a much more effective.

61:04 >> My crappy comfy UI was that like it just swapped out like 10 different versions. Yeah. >> My favorite part I still swap out like I'm enjoy living generate four generate like redoing something just make 10 iterations and >> and then yeah eventually that one's good enough. My favorite is that you pray to R and Jesus because it's a random number. basically you know to me this concept of what is a foundation model is something like this where it's like okay you know all these little workflows that you guys do to like do that and do that and do that and all these different tools we're going to replace that with one model >> and that's that's actually what we're building towards right and that was where flux >> this was a very clear aha moment for me >> I don't know the I never tried prompt to prom but I think to me the closest has come to flux context is Quinn image which like the Quinn folks are doing really good like they had a photoshop layer model. Yeah, it was >> Yeah. No, no, I mean they're doing doing pretty cool stuff over there.

61:59 >> It's interesting how much of this just comes back to it was a lot of good data. >> Yeah, I mean it was it was targeted good data for what is it a good concept. I I don't know what how to even describe us here but yeah I I think one thing that I'm surprised by is so you know comment on stability one more time stable Japan right Japan had this like whole law of like we will protect you if you train on copyright >> I'll open anything up >> as a country and I was like oo stable Japan like it's interesting nothing happened >> I always wanted to visit that office I was sad I never did actually >> I think on your omni point as well it was an interesting time there were the two others Gemini is obviously the latest omni image which I didn't realize is 2023 from Meta. They threw in one embedding heat maps and >> depth. That's a big one >> which you know I'm like oh very early but like it's cool. I I think this work has progressed. There's definitely newer newer.

62:51 >> Yeah. So like I amu >> you know this is just another color right there RGB a whatever that is. >> Yeah. Right. So like how many colors are there? >> Oh the fun thing this was also all all in all out. So you can go text to heat map if you want. You can go IMU to de big market for that. >> But interestingly, you know, no video out, which is I think the the kind of the >> the central thing that that we need to to to to understand basically the world better, right? This this is I think what what's said what has always been at the core of what we we're doing.

63:22 >> Yeah. Okay. Well, before we leave the the sort of data disc or IP discussion, is there a particular kind of data that is everyone in an industry knows is like valuable? if people if if I have have a data vendor listening how do they sell to you what how do they approach you depends heavily on what you what type of problem you want to be solving that's just super important if if I train a a a robot I obviously need a lot of egocentric data I think that's not a secret right because that that is just how this thing will be will be >> egocentric or yeah >> I mean there's also a benefit for you guys right on the language model side there's a lot of people doing our environments is there a parallel that people should come do >> I'm trying to get you vendors yeah it's Oh, we have so many data vendors actually that that approach >> too many. Yeah. Yeah. It's a lot of >> Oh, okay. What what is good versus bad?

64:08 >> It again depends on which stage of the training you are and what your effective goal is. I think there's a there's a trade-off between we had this already, right? scale of the data versus versus the quality. And that is what you have to for every training run like large scale pre-training run you do again you have to kind of find find that sweet spot out because depending on the architecture you you might be using that might look very different again and that is one of the central pre-training experiments like or pre-scaling experiments that you need to do you need to do a ladder basically to understand at scale how much or to to predict at scale how much data do I actually need and how should the quality look like so where's the sweet spot between like these two dimensions of data quantity versus data quality.

64:54 >> Any other major jumps that we want to call out? So flux one context was a big one. Flux 2, flux tools, action, we talked a bit about any other >> I mean I think the three sizes that's very important in terms of like you know the the dev models and the schnel >> yeah what is >> means fast in German. >> yeah I think you probably covered that in the book but yeah I think that that like is you you have open source you have the closed source and you you have the range of performance I think that makes sense. Yeah. Yeah. I mean I guess if like anything to track on is like okay one the one like thing that made me quite sad that we didn't quite get in with flux context is it was only single image edit because I mean I s it was very clear that there was like a huge need for for multi-image like people you know want to be able to put in many many different images and relate reference is a better word here. Yeah. Yeah, many many different references >> which I think Grock is like king right now.

65:47 >> Yeah, I I don't know this. >> I know I know Mid Journey does quite a bit on style books where you can build up >> reference. You mean they've always been very good at like the sty style reference, but this is more just like general like like f flux context would only take one image and so it was just you can only iterate without a bad photo of one person that's but you could have three different photos of the same person.

66:04 >> But it was actually general enough. I remember it was it was actually quite funny. People were like doing workflows where they were stitching a bunch of like images into one image to like emulate it. But then with Flux 2, we finally were able to like spend the time to to to kind of build out the data and actually do this larger scale training for handling I think it was 10 reference images at the time. So essentially building up much more context to work within.

66:24 >> Yeah. >> Yeah. >> Which is again also a lot of to to a large extent a data data problem that you just need a data as these many references. Right. >> Is there a practical limit? So you know let's say the industry standard is like 10 characters or 18 characters. That's it, right? >> Yeah. I don't need 100. >> Obviously, you can show crazy numbers on I don't know how many on how many characters you or like like images you can get into one image or also video, but there are like that is agent work then to to get me 100 reference >> which is the easy thing, right? It's agent work.

66:54 >> But then I need to check it like I I then I have to review 100 images if that is actually what I it's actually also a hard creativity problem like how should I where should I put these things >> 100 images like So I think without like like overindexing on the number everything that goes up like like over over 10 10 references is also hard to follow for a human. I think like if you say okay we want to be creating a a video right I obviously can can put in 50 people there but if you look at the mo like the the most important movies how many central characters do we have >> like five six maximum >> unless you're Avengers.

67:27 >> Exactly. that that is then also it's at some point approaching the the kind of limit of what we humans can actually kind of process in in a 2 minutes clip or something. This also gets way like coming back to the I know you mentioned the evals earlier building evals for larger like contextual models becomes a much bigger process like even for like we we ended up releasing a an open eval set for it was context bench which was built off the internal bench that we used to evaluate context and then as you scale up how many references >> oh I haven't seen this okay >> yeah yeah please contact this was a lot of work but kind of it was actually quite fun to put this together >> make the cat very fat >> yeah I mean it We lit I remember sitting in the back room of our SF office with like a few of the other guys. Like me and Tim were were laughing. Tim Dawhorn is another one of the great great guys.

68:13 >> So like do you just like take some edibles and just go like >> I mean you just you're sitting there late sitting there late at night literally just typing in the like we're just pulling images and typing in the stupidest things. >> I mean that like but it's like real like I don't know. So because >> in the end you obviously need to kind of make one step to make it a kind of what is ground truth.

68:33 >> You you need to make sure that >> you have a ground truth somewhere. >> Yeah. >> No, this is just a benchmark basically. It's just inputs, >> right? But where's the where's the output? >> Where's the eval? >> I mean that is the interesting thing, right? You just ask people. This is a public bench to give everyone like this is the eval. And then you you have two models that you want to be >> Yeah. Yeah. So it's not a validation set in that sense. It's just a email.

68:58 >> This is just Dustin sitting in a room genuinely like 80% of this is that like it it's >> heavily influenced by by our I think by our Italy offsite >> when you do like a Mad Libs like like a combinatorial explosion type thing that is more structured than just random. >> So so I mean now we're do we definitely do more stuff like this. I will say that so one other thing is there were I mean there were certain things we had to like filter out for the public set on here cuz it's just there were some very dumb things in the internal thing you know in the internal set but I don't know I I spend a lot of time I like to see how people actually use our models like and I I'm someone that actually really likes to use our models I think it's very very important that you test and play like really just play I'll say play with your models like memes like me like yeah like use it for what people actually want to use it for like I came like like I said I came from Discord open source communities on this you see people doing the stupidest things, but it's like this is people are having fun with it and it's so a lot of the set, you know, besides trying to cover a bunch of actual use cases, like production use cases, which which are very important, I also want to make sure it's just like the models are fun to use, you know, and so >> good good vibe vibe.

70:08 >> Exactly. Yeah. >> What I think is like I I underappreciated three years ago, which was like, yes, this is memes. Yes, this is like a mid-30 whatever subscription, but like it actually becomes very real work. >> Yeah. Exactly. Absolutely. Because they've done it for fun, then they're like, "Oh, I can do this at >> Effectively also like when you when you automatically eval these these models like there's a bunch of obviously human evals that everyone is doing in the field, but every evaluation will have their blind spots and you you should be combining as many independently as possible to get a >> picture that is as good as possible of like how the model is performing. But that doesn't like remove the burden from you to kind of or it's not a burden.

70:49 It's actually super fun to then sample the model yourself and actually understand where are the strengths, where are the weaknesses like this is this is a core part of developing a model is actually >> constantly sampling when developing it and checking a is it is it where I would expect it to be right >> at some level it's like what you guys care about too right like these are your use cases it's a good way for if you want something to get better like make a little benchmark and see how things perform. No, absolutely. I I think it's very important that the people building the models actually like >> like want to use the model. Like I think this is very important.

71:17 >> There's a big distinction in this and like the microphone brand trying to you know use it for product photos. A very niche use case that they can minmax on compared to removing >> very general. Yeah. I I I think one thing I do doubt or question is that a very well-known midjourney is loop which is the it's discord. Everyone sees like other people prompting you get the four options you choose the four that is feedback for your next model. How come you didn't do any of that?

71:46 >> We didn't do it publicly at scale. So, so every one of >> Isn't it valuable to do? It can be valuable but I think only up to a certain extent and if you then like cross certain limits the signal that you get is pretty pretty like early days we saw these insane potential right of this this this tuning say of of majour's models like V5 or something crazy >> but if your base model just reaches a certain certain kind of level of strength it's actually you don't get these super good examples anymore where you can clearly see the model improving from this and this is also a lot of noise. So there are there are a lot of other ways how you can actually >> I had this debate as well with someone where back then it was very useful because prompt adherence wasn't great right it's pretty judgmental on which one follows the prompt better when neither of them do it great right so you have a lot of preference tuning towards there's very high signal and okay the people prefer this now back to my point a lot earlier like >> if I take the top five models and tell them to remove a bed it's all going to look the same to me the preference there's not that much signal right?

72:48 They're all filling it in pretty good because there's like >> they understand this task very well. So that side of the eval for me is kind of gone. It all it all looks fine. But >> yeah, >> I was going to jump and say that was actually one of my favorite parts of of moving into context is finally it wasn't just like >> I I don't know. It was finally just like okay did it actually just do the task >> and and I really enjoyed getting to these like very kind of clear concrete tasks. One more part of the texture and then we want to go into robotics in action. auto reggressive image gen.

73:18 >> It seems like a big trend with both Gemini and Chad doing it. like is that as big an impact in in your mind as it is in ours? You know what I mean? Like I I it seemed like a very big change because then you could conversationally edit. It is the like the next obvious step from cont a flux context where instead of editing with an image and and text prong you're conversationally working with a with a model >> that I would say is not a a feature of auto reggressive models only it's just based on the data you train on it and like you you can get language capabilities into a diffusion model as well right >> yes >> which we know is is working and I said long term this is where we believe this will this will also go point to and where we're working towards which is to also get more modalities in including this language part right so it's not it's it's not a question of the of the of the algorithm there it's just a question of like what you focus on basically I think both approaches are extremely exciting and have a lot of strengths and a lot of like shortcomings I I would say autogressive models are way more data efficient you can train them way better in data because every like you have for for diffusion models you have this trajectory from data to noise right and every training example you're training on is effectively because you have this continuous path from data to noise in between. It's giving you rise to infinitely many training examples because you can pick every possible spot to dn noiseise from from this trajectory between data and noise for one example. Obviously, the model would overfit if you would only train on on this one example.

74:53 Theoretically, it's just like one training example is giving or one data point is giving rise to infinitely many training examples. Whereas all regressive models can just train on all the tokens in parallel basically and get the maximum gradient from each training example. That's the training side. So there I would say our regressive models are more data efficient. However, at the sampling at sampling time or regressive models have to don't get across that. they have to autogressively create one token after the other because they're iterative in this like like data direction, right? And diffusion models you can just go down and distill 50 steps down to four and by that you can speed that up so much which is on the autogressor side just not possible because you have to auto reggressively generate token after token after token.

75:41 So these are just two two >> wait you make it sound like it's just worse than so so autogressive generation is in that respect it has a limitation but on the other side if fusion has in in the terms of data efficiency at training is also limitation so we we actually I think it's I I like both approaches we've also been working on autogressive models like long long back and both is super exciting I would say >> so on data efficiency auto reggressive wins out though >> okay so scale I guess the one of the other big things that you see having talked to people that worked on both like chachbt and genen is how you can add in the loopl like aentically so models can do agentically you know generate something reason on it edit the image and continue that workflow they can do that >> and this is but this is separate than pipelining in diffusion generation right it's all the one say model doing it where >> as you train that in on a data level you know you can scale that I guess is there a counter to this in diffusion There's no inherent limitation to that you couldn't have a model that is diffusion based do the exact same process that can do all of these aspects like I I had the wrong basically nano banana happened and I was like holy like auto reggressive is a thing but you're saying >> I think am I wrong that isn't >> was I think it is diffusion based I think the auto like it is >> and I don't have like insider knowledge here so I might be wrong but I'm pretty sure it's it has the language component to do like autogressive but it is like I think It's what what we'd call like a transfusion model. So like it's you know it can essentially do both but there's no Yeah. There's no inherent limitation that says you you you have to do it one way or another and it's not like we're there they're they're exploration.

77:23 >> Yeah. Yeah. What I'm getting is that it's not it's not settled consensus and I thought it was that there was a wave of people shifting from diffusion to auto regression for >> I think there's a lot of there there's a lot of exciting research on that end right now going on. Like I will say there there's some very impressive work out there. Obviously, we've seen models work extremely well on that front. The same is going on for for flow matching is also improving. Like it I would say the debate is not settled yet. Like there there's still significant re research on both fronts going on right now. And I think I think it's important to watch both and be you know try both be >> excited about the you know >> finally the same is happening for what as well. We have we are vamps basically which is super super interesting as well. Right.

78:01 >> Yeah. The the last other thing I guess is there's also the diffusion for text right. So there's also some of that. So it's there's benefits. There's of course speed, right? You can >> we see that already like with tax we see it clearly, right? How >> we got diffusion gemma that people can play around with too. >> Will they ever reach parody? Is it just a matter of investment in money and data or like you know >> I would say a lot of it is data.

78:23 >> If if it doesn't work why doesn't it work? My simple answer is like we we reason left to right. We don't reason you know and diffuse into into vision. At the same time you like I don't know we with autogressive models I think a lot of people like to focus on the the the you know step by step of the of the tokens but at the same time it also can parallelize like we we can't just take in a whole book at the same time as well which those models can so there yeah >> and you can think latent >> I totally get it so that's I mean that is the valid question like how come diffusion for text never took off and you know now it's like slowly slightly better but like >> I think it's just like a matter of which approach was was earlier and has been like becoming for language model I mean I think as your point you said if it's more data efficient and the world has been doing a lot of scaling there's a side there that doesn't mean that there aren't pockets where there's use cases that you can get really good things at a certain size the example is like yes diffusion Gemma is pretty good and it meets a bar and can run really fast right if you want that that's great >> yeah I mean on the auto reggressive language front it's it's I think it really comes down to it was simple it was scalable it was there It was like a very I mean it's clearly been a very successful trajectory. A lot of time and and money and research has gone into to making that work. There's right now a lot of very exciting research on the diffusion front, you know, and I and I'd like to separate the notion there's because there's diffusion language is really two camps and I think a lot of people tend to not realize this because there are discrete diffusion language models which I almost don't I don't like to call them diffusion models really and this is where like the diffusion Gemma and like the mercury code like mercury models live which is is more of this like masking style model where where it it jumps from a mass to a token but it's still in the discrete space versus now there's a big wave of research right now going on continuous diffusion language models which those are actually operating as like you know I don't want to say flow matching models but they're they're operating in this like essentially dnoising space so so more like >> this was used for audio if I'm not mistaken >> the audio generation >> continuous >> continuous diffusion >> it's probably used for >> it's for everything visual modeling audio modeling >> my last fun question there is do we have a good way to do like your PO style RL on pure diffusion text models.

80:42 >> Yeah, we we're doing this actually. >> Okay. Okay. >> So for flux 3 we did this and and it has had tremendous u gains that it borrows. Yeah. >> So flux action this is coming to very very recent launches and and work that you're doing. How do you approach this? What is your what is the feedback and impact so far? >> Yeah, it's actually like how how we approach it. Basically you you pre-train on again videos, images, audio, everything together and then you have this very general model that understands the world and then it actually you can can take a bit of action prediction data. Basically, you add kind of predictions in the state space of this robot, which is basically a kind of a parameterization of here is this robot arm. which every joint will be described by a translation and a rotation, six degrees of freedom basically. And then you predict that together with a video. And since the model can already reason about the video and what it needs to create or to to fulfill a certain task in video space, it's actually very simple to then add this new dimensionality or this this new dimension, the state space how we call it, and then basically predict that as well. Given a chunk of earlier video, condition it on that an instruction basically the model creates the next chunk of actions to basically fulfill >> this action. And this this works extremely good. It's we can do it in the real world like for real world robots we can do it in simulation like computer use is also kind of a mapping of that to the digital world and we can like we can make the model play games right and stuff like that. So this is all enabled by this this single flux three model.

82:15 And then you see also that it's extremely capable because we have this very general pre-training which is not only limited to say classic robot data but actually all kinds of data that gives a model just this inherent build understanding that you also see then in performance and being a diffusion model is also pretty speedy as we see here with a with a with these distilled versions basically right. >> Would you call this a world model or world action? And do you have a term you like to use?

82:42 >> It's a world. This this one is a world action model. I think the word the term world model I don't like to use. It's so >> I'm in the same boat. It feels like omni to me at this point. >> Let's call this a visually intelligent model because it can do all these these basic things, right? >> I guess the the distinction that I see in these models is you're not just predicting video or worlds, you're also prediction action states, right? So one is what is happening in the world? What is the state? but also what is the next action to predict. So it's like sort of two outputs, right? So >> is there a target frame rate that you target that is industry standard >> because you don't have to predict out that far. It's my plan. It's my idea.

83:21 >> You mean in terms of the real time factor like how how it's like 10 FPS I think what the model currently runs pretty small >> and you're projecting out like one FPS or five F sorry one frame five frames whatever. >> I would have to look it up. I think it's it's a small chunk of of frames. I would say like in between 9 and 21 frames that it then predicts again and then the model is conditioned on the previous generations again and what the camera when it runs on the robot what the camera actually perceives right and then you basically predict this further.

83:50 So it's it's it's a bit of a chunk. It's not running autogressively so to say in in in time basically but it's a chunk prediction. What I'm driving at is like this video prediction side that is very much for predicting the results of what you do. But so probably you need some kind of long range planner existing outside of that model that will coordinate the actions that >> I mean this is this is why we're discussing the language side earlier.

84:14 >> Yeah. Yeah. So as it runs right now what we see is that there's no plan like like high level planner involved or something. This is just a model running on a on a on a loop. >> Yeah. >> Some of the interesting >> what we did with Dina Robotics. >> Yeah. Dina has one, Runway has one. I guess talk about the fine-tuning side. So, if I want to take your general model, fine-tune it for a new robot with a new task, not something super crazy, but you know, something that's out of distribution, it hasn't seen, say it's my own home robotics, I can program in how to use a humanoid arm or whatever.

84:46 what is that process like? How long does it take? How important is that in fine-tunability? Also again like since this model is so general and then the pre-training was so nice it it takes like compared basically you need a bunch of hours I think we showed one example where we compared it with I think it was power 5 which was also open source right it took eight times less data to basically train on it that was for industry application it was like 4 hours of data that we that we needed basically to get to a success rate that was like good enough for this basic application >> and is that a thing you kind of want to hell climb on right now or is that like >> we go back to the context thought >> we shouldn't be doing fine tuning >> what we what we're targeting and what Bill we're working on right now extremely >> carefully and with a lot of focus in context learning for robotics this is you should be showing a robot in action one time and then it should be able to do this and this would be truly a superhuman intelligence like if you if you look at how long it takes us as children to learn walking to learn using our thumb correctly to grab this cam for instance, right? This is like weeks and even months of of learning that it takes us to do this. Our vision is that robotic like an autonomous robot should be able to observe any action once and then be able to do it.

86:03 >> Mhm. >> That is what would be a clearly superhuman kind of visual intelligence and that is what we want to be reaching. >> What is the standard now? Like how many demonstrations do we need? >> It's around like 4 hours of fine tuning you need at least. >> But even even four hours is pretty good. I I would say 2027 will be the year where we will >> mid training is an interesting one in here too, right? So is this like I want to go from manufacturing to maybe maybe medical stuff like filling pill capsules like specifically like different niches for example is that where >> I mean the model the model is trained quite generally. So even even the midra like before fine-tuning it might be able to handle many of the like even this version might be able to handle many of the tasks you throw at it. The fining is really just nailing, you know, in the robotics world, you want to you want to act you want a certain level of accuracy you expect out of it.

86:51 >> Exactly. This is just a standard plain large scale mid training and then we move on to this finetuning stage where we add the high quality robotics data in >> and then we get this this this further bump. But you see that already to to a to a somewhat good extent on mid-raining the model can actually do these tasks. >> I guess the other interesting thing is it's a 7B right? Is that mostly just constraint from these things are deployed?

87:16 >> Exactly. You want to be having you want to be having a model that is as small as possible because you need to be >> able to act first on device if you want to be adding this to a manufacturing basically and then also in real time >> and this is on the backbone of Flux 3. >> Yes. Mhm. >> And do we so is like the image and audio all that stuff >> all like this is this is derived from this one model basically we have this multimodal model >> that that is really important to understand this has been trained on a huge corpus of images audio and from that we can derive everything that we basically released this summer which was flux video flux video edit flux action flux image flux image client we also derived an upscaler from it we have a super nice 4K video upscaler which is I think super fast also can only recommend it's really good and this is all powered by this one model and this this shows how training these very general models actually compounds then over time because you can derive all these applications from them right >> is there a difference between a video upscaler and an image upscaler isn't >> a video upscaler we talked about the temporal consistency obviously if you would have a upscaler that would have like like no temporal understanding then you would have this flickering again this >> especially in the high frequencies you then real like like you don't see this too much when you comp like look look at two images side by side you don't see that much but if you switch between the images that's that's one of the basic tasks that everyone who's looking a lot of images does just opening two images in two tabs and and then switching left right they see every single single difference and that would just amount to kind of flickering in temporal space if you wouldn't have this temporal component in the video upscaler and that's what makes it difference from a image upscaler >> any notes on real robot evals and how to scale this Like I I see, you know, sponge in bowl three times like and and at some level like if you look at our Dapod, there are literally people just one off testing robots cuz you have to test in real scale. But like come on, two out of three runs on Sponge and B.

89:16 Just restart, do it again, you can be three out of three, right? Like >> it's it's a hard problem. I think one one one thing that's constantly being worked on is just closing the SIM trial gap. And this will get better and better and better over time. But to be honest, evas no matter where you do it, it's a painful thing. It's also painful when you when you look at the content creation evas like ask >> three different groups of voters what a good image looks like and you would get you we you will get three different answers.

89:42 >> You have wars over it. >> We have wars over this >> but that's evas are just for generative modeling. Evals are just hard. >> Yeah. >> Yeah. >> Is this already like how developed is the commercial site? >> We will have >> more. I cannot tell too much but there will be soon for DFL powered robots in productions live and >> 2027 will be the year of robotics. >> Oh absolutely. >> Okay. Say more on predictions. Where where do we go from this in 2027? How how fast do things pick up?

90:15 >> Sing yeah in learning is a big is a big thing. We'll see first we'll see first commercial applications of these things in real manufacturies. That's that's why it's actually good that we have this German base because there's a ton of industry data >> and obviously lots of potential for partnerships. People are really hungry for robotics, I would also say. >> Yep. >> So there's a there's a good bunch of of synergies that we want to be using to actually make 2027 the year of robotics.

90:45 >> I I'm very excited about the research the research on this is also moving very quickly. So I I suspect that robotics will improve very quickly. >> Yeah. What about the side of like robotics is very physical manufacturing constraints like you can scale up tech you can scale up models but that's it's pretty hard to affect right so predictions 2027 where where where are we realistically on robotics can I buy something off the shelf that can single shot learn like >> I suspect like later 2027 like people will be able to get some some quite interesting robotics in their house I mean I whether it's you know is it going to be able to do like fine motor tasks and all this I I don't know but I suspect like and again this is just entirely predictive. I I have no baseline for this, but I I I suspect next year there might be some some interesting like you know at least niche for like the the tech side of people like us, not like you know the average person would be picking this up but some interesting potentially useful robotics in your house >> and for manufacturing I think 2027 will be a year because manufacturing is like more restricted right than than kind of it it isn't that long tail as like say autonomous driving and stuff like that right or or like >> consumer robotics where you can just like do you would ideally like to do this thing to do anything you want but for manufacturing I think there's a very good chance that we will see like a lot of applications actually being unlocked by robotics in 2020 so >> and that's what we are also working towards basically >> very much so >> awesome I mean exciting it seems like very fun place you guys have robot labs video models >> building you're just missing the flying car >> you mentioned Germany a lot but you also have a pretty big SF office I actually didn't know there was a big European side there's a substantial SF presence called Black Forest.

92:26 >> And this like inviting everyone to come over to our place >> 410 Townson Street. super super nice and we we we're not a European company, not like we company with roots both in Europe and the US and we were acting globally basically. >> It's fun. We got you on because you're visiting in town and we're like, "Okay, I should have." >> Yeah. >> How can people help you hiring? >> Yeah. looking for >> I think just just supporting us in marketing talking about our models helping us also here I think in the US getting getting our name out out in the community we're well known >> I think for for like like in the broader society not that much there's a big players right that's also what we need to change in 2027 really kind of getting our name out because we have extremely strong technology I think that that that's what we saw we have a long history of having built like the core pillars of like what everyone is using basically it's now also time to get our name out to the kind of broader audience.

93:25 >> Yeah, but the the tech community I think we're well known but >> apart from that >> tell your mom about flex. So, it's it's very interesting that the overall gen obviously I'm a marketing guy and I help companies in marketing all the time. You guys are very different cuz generative media in general has a weird mixed reputation in the broader population, right? Like normally I would suggest oh have a Tik Tok or Instagram marketing campaign but pretty much you'll be killed in just because they hate AI so much, >> right? And so it's very interesting to find a niches where this is isn't regarded as AI. It's just normal a thing you can do with tech. Mhm.

94:00 >> But it's not AI because I think the moment you say it's AI, the population will just >> have pitchforks. >> I don't know if you have any response or thoughts or explorations about that. >> I think it's important to to just always also emphasize that we we we take this responsibly. We're like like not our models are not creating like avalanches of slop or something, right? They they actually live in real use cases. lots of like professional workflows and they just help making the lives of like professional people who work with this way easier and just increasing productivity so much.

94:34 >> It's Photoshop. >> Exactly. It's so crazy like how how much more productive you are then. And that that's really the the kind of the goal of this on the content side. It's not about putting people out of work. Not at all. It's just like making everyone's life so much easier. >> Yeah. I wouldn't say that. I'm just like if I worked on your team like this is the the my GTM problem right now which is like how do how do we get people to basically you know go viral in the same way that maybe whisper flow goes a little bit viral cuz they're like oh yeah I understand how to use this is not AI it's just whisper flow is granola same thing I think that that there the generous media hasn't had that crossover except for Nano Banana and then whenever Chachi PT had the >> studio Gibli moment yeah >> those kinds of Like that is the crossover where it's like oh like this is a fun trend that I want to hop on.

95:23 >> There were moments that hit well like your point of the see yourself as you're older like that had a huge moment. People would >> look at themselves as their 80s send it to their family. Like there have been moments but nothing is super >> I I think one of the difficult pieces is it's actually very hard to predict one of these like viral moments. Like I like Nano Manana was the like their internal name and then they just kept it because it it like people just like the name and it stuck. for context. Actually, the one big use case that we saw a ton of usage on that like, you know, we didn't necessarily expect was image restoration as well.

95:54 >> So, there was a ton of like I saw amazing amazing examples of people, you know, taking old degraded family photos and then restoring them, >> colorizing them, fixing them up, but but and it was able to keep like a very very high level of of similarity to the to what the original would have been. And then I don't know that that was actually one of my favorite moments that we've had of people showing like their family these kind of restored photos and the reactions from it. And I I really really enjoyed that like positive moment that that it brought. But like I said like we so often we can't predict like what the thing is going to be you know it's it's even with even with our old you know even back in the stable diffusion days it was >> always surprising what the trend was you know online like it just kind of they just kind of emerge they emerge. kind of plan them. I think once you have them, you need to to kind of exploit them obviously and you should be always asking >> what is the next big thing, the next capability that it has not been >> able or like like that models have not been able to do. What would that be? And then trying to to bake this into your model, making bets. Some some of them will work, some others will not work.

97:02 But this is just extremely important to to kind of always have this mindset of hey what what is the next thing we should be working on that will just bring a lot of joy or like productivity increases to people right >> well that is the domain of research you guys are doing amazing job so thank you for sharing your story I hope more people come to know you as a team because they they've seen a lot a lot of the model buzz but I I feel like this is also the first time I've seen I've heard the full story of >> yeah thank Thank you. Thank you so much for having us.

97:33 >> Thank you so much. I mean, this is a It's fun to finally have gotten to do this. >> Yeah. Yeah. I mean, it's a year in the making, but also, you know, I bought a bunch of micro ducks and I, you know, I'm looking forward to them running around. >> Do Do you have like I I want to get my hands on. I feel like I missed the window of these being available. You have connections.

97:51 >> I'm I'm sure I'm sure I Whoever out there can get me a micro duck. Give me a micro duck. >> I'll get you duck. >> Okay, perfect. >> I bought a bunch cuz I know this going to be a hot gift for Christmas. >> Yeah. Yeah. Yeah. Yeah. feel like >> but but yeah you know robotics 2027 that's you know one year away like you know we we'll see we'll see where this all learns but I'm very optimistic about this approach and I think you guys have explained it very well.

98:12 >> Fantastic. Yeah. Amazing. Awesome. Thank you. Cool. Thanks. All right. Thank you.

Summary

In this episode, the founders of Black Forest Labs, Andreas Blackman and Dustin Podell, discuss their journey from developing latent diffusion models at Stability AI to creating innovative generative models at their new company. They delve into the advancements in image and video generation, the importance of data quality, and the future of robotics, emphasizing the potential for AI to enhance creative workflows and automate tasks in various industries.

- Black Forest Labs was founded by former Stability AI researchers to further explore generative AI, particularly in image and video generation.
- The company has developed several models, including SDXL and Flux, which enhance image generation and video editing capabilities.
- Latent diffusion models have revolutionized generative AI, allowing for efficient training and high-quality outputs.
- The importance of data quality and quantity is emphasized, as it significantly impacts model performance.
- Black Forest Labs is expanding into robotics, aiming to create models that can learn tasks from minimal demonstrations, potentially transforming manufacturing.
- The founders believe that 2027 will be a pivotal year for robotics, with significant advancements expected in commercial applications.
- They highlight the need for responsible AI use, focusing on enhancing productivity rather than replacing jobs.
- The conversation touches on the challenges of public perception of AI and the unpredictability of viral trends in generative media.

Questions Answered

Who are the founders of Black Forest Labs and what is their background?

The founders, Andreas Blackman (Andy) and Dustin Podell, have a rich history in AI and image generation, particularly with stable diffusion. Andy co-invented latent diffusion, while Dustin transitioned from a freelance web developer to a model and research lead at Stability AI before co-founding Black Forest Labs.

How does the diversity of backgrounds contribute to the strength of Black Forest Labs?

The founding team comprises individuals from various fields, including hardcore academia and the open-source community, which enhances their collaborative strength and innovation.

What are the key improvements introduced with SDXL?

SDXL introduced significant enhancements over previous models, including multi-aspect training, improved aesthetics, and the ability to handle various image resolutions, marking a step forward in the evolution of stable diffusion models.

What is the significance of flow matching in diffusion models?

Flow matching is a mathematical approach that allows models to translate between different probability distributions, making them more scalable and easier to work with, thus enhancing performance in generating images.

What challenges exist in achieving controllability in AI models?

Controllability remains a significant challenge, particularly in maintaining character consistency across models. The development of Flux Context aims to address these issues by allowing for more nuanced prompting with both text and images.

How should AI models be evaluated for performance?

AI models should be evaluated through a combination of human assessments and automated evaluations, with an emphasis on continuous sampling to understand their strengths and weaknesses during development.

What is the process of fine-tuning AI models for specific tasks in robotics?

Fine-tuning involves adapting a general model to specific tasks, requiring significantly less data and time compared to training from scratch. The focus is on context learning, allowing robots to learn tasks efficiently from minimal demonstrations.

© transcribe · For agents Built with care and craft by Gokul Rajaram