Fireside chat on everything inference

0:00 Hi, I'm Tuhin. I'm the CEO of Baseten. today I'm here with My name's Amir, CTO of Baseten. Hi, I'm Bola. I work on product here. Yeah. So, you know, we've been building inference products for the better part of five and a half years now. is that right? Five almost six almost six years, which is wild. you know, obviously the market's evolved a lot in those six years. I mean do you want to just start start off with you have a little spiel where you talk about you know where we come from and why why we're doing what we're doing. History lessons. Yeah. so yeah the company is almost 6 years old. the product has always been in the ML infra space and within that it was always focused on inference as in we stay agnostic of training the models. We cared more about what comes next. the thing that changed were the types of models that customers were deploying on Baseten. Early on it was classic machine learning, predictive models, regressors, classifiers. and the use cases for those were mostly back office use cases, fraud operations, trust and safety, content moderation. But about three years ago that started to change. We started seeing a lot more customers deploy deep learning models, BERT-based models. And that was exciting for for two reasons. One is that these models were suddenly being used in the path of the end user of our customer. So suddenly the customer cared a lot more about high throughput, low latency, SLAs, uptime, things that we were building for, but things that the back office people didn't care as much about. And the second reason it was exciting is that we saw a pattern that these models, these new architectures, the models were becoming useful in ways that you couldn't predict. and often times their weights were open too. and so we said look if this continues down the line we're going to end up with models that are both very useful and open in different modalities. And so we decided to build for that future and that meant building a lot more depth of product in really two pillars. One is around optimizations at the model level and the different techniques there to get the most out of the hardware and then the second time around horizontal scale infrastructure and cross cloud and built a lot more depth there and that was good timing because by the time two and a half years ago came two years ago came and a lot of companies were using these generative models in production we had a good product for them and started signing them up. Yeah. And so yeah and we've always focused we've we've always focused on this problem from what like three pillars you'd say right there you said two but there's a third one right so that's right so the pillars performance performance at the model level infrastructure but really these things have to be wrapped around in a very good developer experience that presents the power of these two pillars to the customer so that they can optimize and and get the most out of their the the models for their own use cases because the use cases are very different based on modality, based on whether they're latency sensitive or throughput sensitive. Yeah. And and so you know we we you know in I'd say like there was like the ChatGPT moment. I think actually what was more interesting in a lot of ways was the stable diffusion moment because it was all of a sudden you had these open source models which you know could you know be considered state-of-the-art and Wispr and and Wispr and I think you know there's always been this debate around open open source models and closed source models and custom models and whatnot. We I think we're we're probably a little less dogmatic than others. we kind of think that the future of AI is both open and closed. And what we're really focused on is that look, there's going to be a ton more models. you know, we're probably in early first we're pregame in terms of where we think we are in terms with AI adoption and these models are going to have to run somewhere. And that's you know in a lot of ways it's like you know we've been very very focused on inference which you know I guess I was going to ask you actually can you maybe bowl this one's for you can you define what inference is it's not for tourists I can tell you that inference not for tourists yeah inference is essentially like how you take a model that was trained and the way the model is trained is just to repeat everything and get it to do something interesting right you get to do like like it's not like training you just repeat like your goal is to mimic but but inference you could take something that was trained to mimic and teach it to generate you could teach it to generate different types of things and all of that is an inference. Yeah. So it's the actual generation of one of the large models and when we talk about inference you know we talk honestly about all the things it takes to actually generate those things in for for users for end users. and so one of the one of the north stars of our company, has been, hey, we're going to build whatever it takes that's going to lead to more inference.

4:52 and I think one of the challenges from the outside in, or one of the perceptions from the outside in is that, oh, this is actually a really simple problem. You take a GPU, you map a runtime onto it, and voila, you know, your model's running. But are you saying that VLM plus H100 equals inference? Yeah, bas basically. But you know when we but when we go and talk to our customers and when we you know when we're running our workloads in production there's so much more than that goes into it. I mean do you want to just talk very quickly about you know you know the two different things there that we've talked about just go a bit deeper. So there's performance and there's infrastructure infrastructure side. Yeah. but but go to to go back to that equation and this is interesting because I see this in the market where where people think that you know they have some GPUs and you know put their models into VLM and you know that that is production grade inference and and we know that through experience to not be true. I I think with those you can get a 80% of what what you want the model to do and like 95% reliability. but for mission critical inference that's not enough. and and you need to go you need to go much beyond that and get closer to that 100 100% number. So so what does it take? It is it is the the what is the inference stack exactly? And so like honestly like this is why we built the Baseten Inference Stack is to get you to 99.99% reliability is to get you to have the power to get your models to do exactly what you want and be optimized for your use cases. so what does it take to do that? So there's really two two components to it. there's the the inference runtime and then the inference optimized infrastructure that that runtime runs within. So so let's talk about the the inference runtime. there is a a lot more that that you have to do than what you can get out of the box using open source. So so you need to have a lot of different runtime optimizations. underneath that you need to have a lot of decoding strategies for that work really for your use cases whether they're latency sensitive or or you know extremely high high throughput. and then a lot of kernel optimizations that that also need to happen and in order to for for you to really get that sort of SLAs's that that you want at at the end of that you have a model that is running on that one GPU very efficiently. and that's great but that's not enough because that one model running on that one GPU very efficiently is going to get too much traffic that it cannot handle and at that point you need to horizontally scale it.

7:36 and that's not an ML problem or a PyTorch problem. That's an infrastructure problem. So that's where the second pillar of the Baseten Inference Stack comes in which is inference optimized infrastructure. so what does that do? So really very high level three things. you need to horizontally scale up and and do so very efficiently and very fast. There's a lot of cold start work that needs to go into that to ensure that when you get a burst of traffic, you can quickly scale up. that's great. Then you have routing strategies that you need to apply and and and and you need to ensure that that that's being done efficiently. so for example for a lot of code generation just doing a a round-robin routing is not going to really do you well and you need to do KV cache aware routing GPU aware routing etc. The third thing underneath all that is to get capacity where you can to go crosscloud, cross region, and be able to gather capacity from anywhere that you have access to GPUs, and not be confined to just a one cloud or one region kind of strategy. combining all of these together then you can you can get the models to be optimized specifically for your use cases and be able to get to four 9s availability. Yeah, I I I think you know just to put like put it in some customer context we we have had a customer where they've been optimizing for latency. they've been optimizing for latency and you realize that this like the inference stack goes just a lot further than just like hey can I just you know run in a highly efficient runtime so I'll give an example here so this customer they fine- tuned a model they needed to deploy it somewhere so and they had multiple versions of this model because they're constantly updating it so they needed a workflow to be able to deploy that model they needed to observe how well it was performing so they needed an observability stack that's on top of it they needed to use prompt lookup which is a speculative decoding method to get the most performance for their use case. and then they needed to geolocate all the hardware close to their users so that they could you know minimize the the hops to to totally for certain use cases where like tens of milliseconds matter right so like if if you're typing if an end user is typing and you want to suggest something in a code editor tens of milliseconds matter if you're doing an AI phone call tens of milliseconds matter and and so that's where all these different tactics and and strategies come into play and And then the inference stacks really becomes the kind of the combination of those three things, right? So it's like at the at the lowest level the bedrock infrastructure. then on top of that you build the runtime and and there's a lot of work that goes into runtime to allow all the different you know prefill and generation strategies that to to get things as fast as possible and you know that's when you can use stuff out of the box like VLM or SG lang or TRTLM that we you know that you can run on Baseten or you might have to do something a bit more custom. So for example, you know, for embeddings, you might want to need need to run a hyper optimized embeddings runtime for which was, you know, we've built, you know, BEI. but then on top of all that, you need the actual develop developer experience. So you can give that control to do all those things back to the user and that's what we've built, which is the inference stack. and that that has become the bedrock of this of the dedic of what Baseten is today and why it you know supports companies like Descript and WRITER and Abridge and Bland to to power things in production.

11:11 one we we've dealt with this we we've kind of delivered this this stack very much in the context of I'm a user I have models I want dedicated I want dedicated hardware and I want dedicated deployments yeah generally for their own custom models that they've pre-trained or fine to yeah I guess one one thing I want to say is like I'm glad you brought up like AI phone calling and I'm glad you brought up embeddings cuz like the basics is not limited to LM like it's not limited to tokens so like for voice for example the routing is different because you want consistency across different generations of the same voice we do the same for images where it's relevant for transcription where it's relevant depending on how the model is implemented so I just wanted to be very clear that like Baseten infrastructure allows these like compound AI use cases because it lets you become optimized for each part of that system yeah and a lot of them require websockets and just the REST call is not enough which also support it yeah there's a ton that's going on there you know, one thing we've we've heard from customers time and time again is, hey, like I'm not I'm not I'm just getting started. you know, I've been using OpenAI, I've been using anthropic, I've been using the model providers. you know, how do how do I transition from that to, you know, my my own dedicated stuff without with the right cost performance trade-off and what not when I don't have scale just yet or I'm still experimenting with scale. that's, you know, where the model APIs that we're launching come in today. we're really excited about this.

12:47 It's been a bit of a sprint for the company to get these out over the last few months. but do you want to just talk a bit about kind of the motivations for the product and what we're launching today? Yeah. Yeah. I think you guys talked about like we've been around for like almost 6 years like like why now? Like why are we doing this now? And I think something very exciting happened in January and you probably heard a lot about Baseten around that time which is Deep Seek dropped and and when Deep Seek dropped like not only did it move markets but like there's something that that was very obvious to us cuz we've seen a few of these shifts like in our lifetime like like AI shifts like just within the lifetime of the company which is like hey there's there is like a very clear open- source competitor to like quality that you could only access through closed source. Mhm. and that became very clear to us and there there's a bit more to say about how it was trained that we'll get into later.

13:38 and that was really like why now is that we could finally put up a model API that gives you a clear alternative and a good place to ramp up and try new things where you're not sure if they will work where you need like the generality that you could only get through kind of closed source before. And we want to make sure that everybody can access that and everybody can get like a world class experience trying to build their products on it. Yeah. And so and what what do you think those things are like as as you're using these model APIs? What are the important things for customers and what Yeah. Like like you're trying to build a product, right? So you want to be reliable.

14:13 You you actually kind of want to be predictable like you want intelligence but you don't want like unregulated intelligence, right? But so a lot of that is like you need structured outputs, right? like you need to be able to sure that the next system that's going to consume this response like has a very clear directive what to do. So there's structure calling, there's tool calling. there's just knowing it's not going to go down. Like it's like nobody wants to be in the middle of a demo and they just be like oh sorry like let me let me let me look at this page. You know it's like just you go to the status page and there's a Christmas tree of you know green with lots of red around it. Yeah. And like you just don't want that to be your status page, right? It's like it's like you just want to make sure that like you're you're insulated from that and and honestly like just just performance like like like it needs to be fast like everybody knows like GPUs were a critical part of the AI journey like because it was the only way you could do things fast enough for it to be relevant right and then that just keeps going through any experience that you want to build on top of AI and like that's a key thing for us and like why we're we're spend so much on that inference stack and like making sure that that's really what powers like this this experience. Yeah. And you know we're launching our model APIs with a bunch of cool partners. do you want to talk a bit about that Paula? Yeah. So actually root Retool is bringing open source models to their platform for the first time on top of a stand like again like I mentioned a lot about what deepse enables and and some of these newer models being able to compete in the open source stack. And we're exciting to continue like working with partners to bring every modality like every model that you might need to build the kinds of products that we know our customers are building. We're hoping all of those will be accessible through the API for sure. And then may maybe so we don't bury the lead. It's you know available today performant comparatively priced and we'll be constantly updating the models available there to keep pace with market. I ideally even be ahead of market. awesome. So that's the first part of what we're launching. The second part that has become more and more important to us as we have gone through you know built this product and heard from customers is you know one thing I said earlier was you know we will build anything that will you know eventually lead to more inference you know and and you know we've historically stayed away from training over the last like six months you know we've got lots of inbound from customers which is very much that hey you've built this amazing inference experience you I also have workflow for training. Is that something you can spot? Hey, I care about fine-tuning. Can you help me there? yeah, Paul, can you do the honor to tell us a bit about training and you know, you know, why we're building this and what we're releasing today and who we partner with. Yeah. So, so over the last three years, right, like the use cases that are really at scale that like really need Baseten need something very specific. So, air talked about like some latency things that people often need. like there's there's reasons why you need to have a dedicated deployment and like we love doing that, right? One of those reasons is that an off-the-shelf model cannot solve your use case. and and some of the most successful use cases on Baseten today are fine-tuned models. and over the last 3 years like our customers are asking us like hey like you guys solve infrastructure for my inference like can you solve like can you just like we had a lot of pain like getting to this point till so till we have this fine tune model that we could then not have pain in scaling like can you just help us like just just help us solve. Yeah they would they would SSH into the box and write some fine tuning script. Yeah. Or like or like they need commits for a year to like try out something new, right?

17:58 Because like the the compute you need is actually like quite large, right? and again like it's been 3 years. So the question is why now? Like why why are we doing this now? And and going back to January, right? Like one of the things that was very interesting about DeepSeek was how it was trained. and and this new technique called GRPO. was that was it quiz time? GRPO stands for group relative policy optimization. well, we can break it all down, but I'm just going to TLDDR like like what what GRPO is. But, I think one of the key challenges with fine-tuning that that people have faced is like what is the amount of data that you put into it? Like how does it need to be structured? What does it need to have? Like there were a lot of vibes like on on like like what data leads to good models and GRPO just said hey like let's let's like let's let the model deal with the vibes right so like so you tell me what you want your input to be you tell me what you want your output to be and the model is going to generate anything in between to get you from that input to that output and all you need to do is tell it here are the right signals that you need to hit that we're looking for for you to be successful right you could tell it some things are good, some things are bad. So these are the policies or the reward functions that you're optimizing. And the way you get there is like by generating a bunch of candidates in a group. The model will generate a few candidates in a group. And then like you will score them relative to each other. That's the relative part with your policy which is these reward functions and then you optimize. That's all the words. That's all the words. and you know we kept hearing about this from customers. so what what are we building? What have we built? What are we releasing? Who have we partner with? Yeah. So so we've built the infrastructure solution that solves like all of the pains that you might have. So like one thing that that our customers have said is like hey we actually need multiode training cuz these models are getting bigger and like even though we could do adapters and we could limit how much like memory we need like we still like we have a lot of data or like we might need to to deal with a lot of data like we need multi node.

20:16 We have multi-node and we have it on demand like as in like you could kick off different jobs that need different amounts of multiode on Baseten that that again like don't have to run for a year like you don't have to pay for them ahead of time to get that year right like like you figure out like what is the experiment that you need to run like like what is the code that you need to run that experiment and we'll solve everything downstream of that so that includes like the infrastructure like the job orchestration it includes the caching of the models and the weights and clusters repeated experiments don't take as long as starting from scratch. we take care of like automatically managing checkpoints and like giving you inference from those checkpoints.

20:55 So I'll kind of just take a tangent here just cuz this this part's this part's important. This is actually cool. Yeah. everybody has seen like, oh my loss curves are cool, my evals are bad. You do a few iterations, you're like, oh my my like my training loss is good and now my eval is good, right? Like my accuracy is good, my eval loss is good, like everything, right? And then you take this checkpoint and you deploy it and then you're like, "Oh my god, my product broke." Like what actually happened? and and I think like this is a common experience and and and part of what is painful there is taking that checkpoint and like going through all the pains of getting it into inference and then going through all the pains of making sure that that's integrated to your product and like do there. So what what we have is like you could take a checkpoint and with one click get an endpoint that is ready for inference that you could AB test in your staging in your pod in your dev and like get real world feedback as to like is what I'm trying to do with my fine-tuning yielding like kind of what do they call it the outcomes that I'm expecting or is it moving the bottom line? Sure. Yeah. That's that's that's what every like it's like and for you like as a product person that's like engagement on the product it's AD testing it's it's everything right and and monitor the performance characteristics around latency and and and time to first token so we build a training API that integrates very nicely with our inference with the Baseten Inference Stack it's available today can you talk a bit about I think we've partnered with is it oxen it's oxen and mixread yeah so so Mixedbread again like we've built this product for post training But Mixedbread is a foundation model company that builds embedding models.

22:38 and we just we just wanted their feedback, but then they're like, we want to use this to train our models. So, we've partnered with them to to make sure that that happens and they have the best possible experience and can get to that value faster. and Oxen, as I mentioned, like like specializes in working with the data and how you version your data sets and everything. and we've partnered with them to make sure that you could get access to these best-in-class compute with these best-in-class techniques without having to start from scratch. Amazing. yeah, we're very excited. So, you can check out both the u model APIs and the training product today. we're really excited to hear what you do with it. it it integrates nicely with the inference stack. you can reach out to me directly at tuhin@baseten.co or amir@baseten.co and bola@baseten.co. if you have any feedback yeah really excited to see what you do with it.