transcribe

How vLLM Became the Standard for Fast AI Inference | Simon Mo, Inferact

Lightspeed Venture Partners · 26m · transcribed Aug 2026
More from Lightspeed Venture Partners Business
𝕏 Share ▶ YouTube 📥 PDF 🤖 .md

Section Insights

# 0:00

Introduction to Inference and vLLM

What is the significance of vLLM in the current AI landscape?

vLLM is positioned as a leading inference engine that supports a wide range of AI applications and hardware. It is built to serve the needs of users deploying virtual machines (VMs) and aims to become the best inference engine globally.

  • vLLM is crucial for inference and runtime in AI.
  • The shift towards using data centers for inference compute is significant.
  • The goal is to create a unified inference engine for diverse models and applications.
# 5:12

The Value of Inference

Why is inference critical in the AI model development process?

Inference is where the real value of AI models is realized, as it allows for the application of intelligence in practical scenarios. Efficiency, speed, and throughput are essential for maximizing the benefits of AI.

  • Inference delivers the intelligence created during model training.
  • Maximizing efficiency and throughput is key to effective AI deployment.
  • Speed in processing can drastically reduce the time needed for complex tasks.
# 10:25

Community and Collaboration in vLLM

How has the vLLM community evolved and what role do hardware vendors play?

The vLLM community has grown with contributions from companies building models and optimizing hardware. This collaborative environment fosters innovation and allows for shared benefits among participants.

  • The vLLM community encourages contributions from various stakeholders.
  • Hardware optimization is crucial for enhancing vLLM's capabilities.
  • Regular meet-ups help strengthen community ties and collaboration.
# 15:37

Building an Inference Platform

What is necessary to create a comprehensive inference platform?

To build a successful inference platform, additional components must be integrated with vLLM to achieve significant efficiency gains. The focus is on open-source principles and a culture of innovation.

  • An inference platform requires more than just vLLM; it needs a robust ecosystem.
  • Open-source values drive innovation and collaboration.
  • Cultural values of openness and being opinionated guide the development of Infra.
# 20:50

Funding and Future Directions

What will the recent funding enable for the company?

The newly raised capital will support talent acquisition and computing resources necessary for advancing inference technologies. This funding is crucial for adapting to the evolving landscape of AI applications.

  • Funding will enhance talent and computational capabilities for inference.
  • Inference is transitioning to more complex, agentic tasks.
  • The company aims to become the leading inference engine globally.

Transcript

0:00 VM nowadays become synonymous with inference and inference runtime. I've been reading all the news about data center build out and new capacity being added. The way I'm reading it is all of those will become inference compute. The cluster are being built for training, but 6 months later, the cluster will be used entirely for inference. We're seeing the paradigm shift from building vertical system. For example, OpenAI there's inference engine for Sora, there's another inference engine for new version Sora. We're building one inference engine for all the current and future model, for all the current and future hardware, and for all the current and future applications. Our end users are all users of VM. So, that means regardless of anyone, as long as they are using VM, deploying VM, and need our help with any of their services, and within their hardware, within their model, >> >> we are there to serve them and want to help them to make them successful. So, that we can all together make VM the best inference engine in the world. Hi, everyone. I'm Bucky Moore, partner at Lightspeed. And I'm James Alcorn, also a partner at Lightspeed. And this is the Investment Memo, the show where founders reveal the stories behind their businesses and how they will shape the future of their industries. Today, we're joined by Simon Moe, CEO and co-founder of Inflection, the company making fast and efficient AI possible with a high-performance inference engine called vLLM. Simon co-created vLLM as an open-source project at UC Berkeley just over 2 years ago.

1:21 It's already become one of the most widely supported AI infrastructure projects to date. Major cloud providers and leading model labs use vLLM in their products, and Inflection is going to deploy it at an even larger scale. As a partner investing in AI and infrastructure, I co-led Lightspeed's Inflection investment alongside Bucky. And I'm privileged to have known Simon for a few years before the company got off the ground. So, I'm really excited to share this story with you, and today we'll talk about how inference became one of the most valuable layers in the AI stack, how Simon grew such an active developer community around vLLM, and Inflection's plans for the future of enterprise AI. So, Simon, welcome to the show. James, Bucky, thank you for having me.

2:10 So, congrats on the funding announcement today. What I thought we could do is share this investment memo with you, which is actually internal to Lightspeed. Have you ever seen one of these before? No, first time seeing an investment memo. Let's take a look. So, at the core of every investment is a thesis, and we wanted to walk through ours with you. So, what you see here is the first point is is this view that we have that inference will capture disproportionate value in the AI stack.

2:30 It's increasingly the choke point where all model execution, optimization, and data flows converge. vLLM is obviously at the center of this, and so, could you bring us back to the moment before you started vLLM with your with your co-founders, and and and what insight did you have that that led you to start the project? First of all, thank you for having me. Going back a little bit, my experience with open-source project and working with Ion Stoica, Joseph Gonzalez, and overall UC Berkeley has been the history of open-source model serving. At that point, we don't even call this inference, but rather it's called model serving for machine learning models. So, that's around circa 2017, 2018. And then after that, we get into into the phase of accelerated computing for production models. And around 2023, while I was at Berkeley, I met Usock, I met Drohan, who are the initial creators of vLLM as academic project. So, at the early days of 2023, they were building a demo around at that point a model called OPT-175B from Meta, and that was kind of a replication, open replication of GPT-3.

3:38 And at that point, they found it very, very difficult to serve that model that big efficiently on accelerated computing devices. And over time, this experience built up. They started to find out different scheduling algorithm, memory management algorithm, and build up an academic paper around it. Interestingly, that was during my one of my first spring break as a PhD, so I got to involve a little bit with the project, but I was like, you know, I'm going to explore something else. So, actually, that's my first interaction with vLLM was a miss. I actually didn't end it up joining the academic paper.

4:12 But very quickly, I joined the open-source project where we turn a academic prototype experimental testbed into a fully functional end-to-end system where anybody can contribute to, and install and run. And we kind of hit the ground running around the summer and fall of 2023, and the story ever since is then become this kind of industry standard layer for inference where models meets hardware, and when workloads are able to use this open-source inference layer to be able to actually run it at maximum efficient speed, and folks are deploying it across the industry. And we talk a lot about speed when it comes to inference. Like, tell us a little bit more about like what it means to push the field forward.

4:52 Is it is it speed? Is it throughput? Is it efficiency? Like, how do all these things come together? So, in the end, it's about getting the best out of the hardware I have right now. So, this is the most expensive part of AI right now. The chip itself just cost significant amount of capital to procure. But then, if you think about the whole process, in order to produce these models, we have to build the data centers, and then build the training cluster, and then from the training cluster, you produce a model. But at that point, there's no value created. Only delivering intelligence and delivering inference is a point where you can actually leverage and capitalize and be able to get benefit of this intelligence. And then the requirement pretty much stem from that. You want maximum efficiency. But on top of maximum efficiency, it could mean maximum throughput. That means processing as many records and as many conversation concurrently as possible.

5:49 But also, it could mean speed and see. It means it could be sinking, having a long conversation, multi-turn conversation, sinking traces, solving a very hard problem, and turning from what would take 8 hours of human labors into an hour of AI labors to a minute of very fast interaction with the external environment. So, speed matters, throughput matters. In the end, cost is a fundamental factor we're optimizing against. And this notion of paged attention, like why start with this? And talk to us a little bit about like how this came to be and the opportunities that unlocks. So, the core technique of paged attention actually goes back to figuring out the transition from a batch synchronous, what we call micro batching serving, to a continuous language modeling kind of serving paradigm. So, before before 2022, I was working on production serving in terms of micro batching. That was when we were serving regression models, serving recommendation models, serving image vision understanding model. What you do there, input is uniform.

6:58 Everybody have similar images, similar size. Output is uniform. It's all the same structure of prediction. So, it's very easy to manage. You take a batch, and then you process it, return it. But for language model, input is non-deterministic because it come from user. It could be a single word hello. It could be a paragraph of request. It could be several document combined together. It could be even just a whole book or thesis for folks to digest, for the model to digest. But then, the output is also non-deterministic. It could be again one word to thousands of word to maximize the entire context window. So, with this kind of non-determinism and non-uniformity, you actually need to have a better way to schedule request and manage the state associated with the request. And that's why paged attention was born, and paged attention is the one of the core technique within vLLM to manage this disparity and be able to maximize throughput of the whole system while maintaining the state, maintaining the utilization, and be able to deal with this kind of different kinds of requests having different lens problem. What is the future implementation of the attention, you know, approach a year from now? Like, how does it change? So, attention has actually evolved quite a bit, even though everybody's saying, "Oh, transformer hasn't changed since Noam and team kind of invented it." But actually, attention has been changing every 6 months. So, in the beginning, we have what you call multi-headed attention, and then later the evolution to make it more efficient has already got underway in 2023 and 2024. That's when we have what we call group query attention and multi-query attention. So, that actually become a lot more efficient already. And in vLLM, we dealt with this variant from day one. But then afterwards, there is attention introduced, I think in the first version of open-source model that will be introduced by Mistral called sliding window attention, which of course goes back years in the literature of academic work, that will compress and only use part of the context window for the next prediction. And then of course, nowadays, if you look at it, DeepSeek has MLA, Qwen has different version, and then Qwen models and GPT-4 has everybody's attention is a big variant from the original intention. And then, looking forward, we're seeing linear attention and different kinds of hybrid, even quadratic attention, and super quadratic attention coming up. So, it's a rich field of innovation. Paged attention is just one small core technique. vLLM today is encompassing tens and even more different kinds of attention variant and implementation.

9:42 And a lot of that, I guess, has come from the amazing community you've built around the open-source project. I wonder if you could take us back a little bit and tell us about, you know, the inception of the open-source project and, you know, the the enormous growth that it's seen since. Like, what did you do to catalyze that? So, a lot of it were I would say we're fortunate and lucky to have that community and to grow with it. And in the beginning, what we have is an open-source project with just passionate adopter contributor. People actually see value in it. In the beginning, vLLM was used as a replacement for Hugging Face Transformers pipeline, which was a different paradigm of running inference because it was really used for training system, not inference system. Then, over time I was seeing folks kind of come around and start to center around the vLLM community. So, we start to have companies who are building models, are actively working with us, say in contributing and maintaining their model implementation as well as techniques. Hardware companies were able to start optimizing for specific models and across the entire variant within vLLM, adding new kernels, adding new supports. And then finally, application company who are deploying vLLM as their for their customers or for directly for the end applications, now start to add all kinds of authorization they believe essential or useful. And all of they want Everybody wants is the same thing, to contribute what they have into the open platform and ecosystem so others can benefit and they can also benefit from others' contribution as well. And we ended up leading the project find UC Berkeley as graduate students to land this entire community and grow with it.

11:24 We have an in-person meet-up every 2 months and this where we kind of see the community face-to-face, but also there's also global community nowadays across the world where different kind of organization are leading to be able to build this presence together. Yeah, I'm glad you brought up hardware because it's such an incredibly important user constituency for the project. So, maybe you could tell us about some how you've engaged the hardware vendors and what your kind of strategy and relationship there has been.

11:56 Yeah, this has definitely been evolving as well. one thing that doesn't change is we want to keep this extremely open-minded. So, that means we are very much receptive and open to new hardware variant. The reason is openness brings innovations and people building on top of each others' work as well as be able to differentiate and based off different workload and be able to understand pros and cons of different chips. So, when we work with hardware vendors, the format and interaction model are quite different, right? For example, for currently the market leader Nvidia, Nvidia has a dedicated team right now working on vLLM. They are focused across different generation of chips as well as chips are not released yet, as well as a software layer thinking about how do the entire CUDA ecosystem integrate with vLLM and together. And AMD have similar thing. For example, AMD have a dedicated team not just working on vLLM, but also sponsoring and supporting the CI system.

12:59 So, AMD has been hard at work pushing, making sure every single test in vLLM when it passes on Nvidia chips, it passes on AMD chips. So, that the users of vLLM can seamlessly switch between. And of course, you could say this is a quite a good strategy from the AMD side and we're happy to work with the entire ecosystem to bring that together. And these are just two examples. We'll work with TPU, AWS Neuron, Intel, and so so much more. And when did it become to you that vLLM was the de facto standard the way we perceive it to be today? When new model vendor start to talk to us, it's like, "Hey, we want to release a new model and vLLM is our first and sometimes even only place to go. That we want to make sure that the world can use our model."

13:45 There's other ways they can release a model to the world, but they have this collective understanding and consensus from the model side that vLLM is where they should do it and they should build relationship to build it out for. And likewise for hardware, if you're making a new chip today and after you finish the design tape out, have all this understanding of the chip, when you need to build a software stack, not just as a chip provider, you want to talk to vLLM. What I'm hearing is the customer, the prospective customer of the new chip and accelerator are asking, "Can you please add vLLM support?" vLLM nowadays become synonymous with inference and inference runtime so that I'm hearing everybody is asking about vLLM support equal to the way I'm going to interact with this model is going to be through vLLM. And that's kind of the moment.

14:37 So, Simon, we originally met at the Skydeck in Berkeley and spent many hours walking around that beautiful campus. At what point did you know that you wanted to start a company? Yeah, go Bears. I wasn't thinking about starting company when we very first met, but over time we're constantly reevaluating and thinking about this. It's actually not an easy decision, James would have known. This has been a I would say two two years in the making. So, every time we are assessing the current state of vLLM, the current state of ourselves as co-founding team, how we're approaching this, as well as the market. So, I would say it is only until around this time we have decided the market is right, the stack is right, vLLM is at the right time. It also necessitate us to have this company to be able to push it to the next step and further.

15:30 And this really draws on my experience last year spending some time at Character AI running the inference stack and understanding really real-world inference at scale. Right, as much as I know about vLLM, there's actually so much more you need to build around to constitute as an inference platform. And additionally, for the things you build around it, you can actually integrate with vLLM really well to deliver the 2x, 3x, and even more efficiency gain that you can unlock. And you just cannot unlock it with vLLM as a stand-alone open-source software. So, that's the sort of pri- one of the primary realization. And this realization really echoes among the co-founder working at different places, adopting vLLM, using vLLM, pushing vLLM at scale that help us really understand come to the consensus the timing is right, the market is right, there is so much to build, and it is time to do build that. Which set of cultural values did you and your co-founders or do you and your co-founders want, you know, Infra to really be defined by? The cultural values we're really aspiring to start with openness.

16:40 Open source is at the forefront and what we care about. And we truly believe open-source model, open-source implementation, open-source runtime system is the fastest way to bring innovation and to progress. So, as everyone working at Infra right now, we have the attitude of being open-minded. But of course, open-minded doesn't mean accepting everything. There's a clear line we're walking here. That's why the second cultural value is also to be opinionated, to understand the future by understanding the history, understanding the data. So, this is where we want to hear as many things as possible, perceiving as many data points as possible. But then, the open-source value is also about really making big bets of the future, build it out together, and then aligning the entire community to go with us. What do you consider most when you built the initial team? Like you had this group of co-founders that you were working on the project with. I know you're doing a lot of hiring right now. but to to James's question about culture, like how is that manifesting in in terms of the people that you're bringing together to build this company with you? We are looking for intellectually deep, right?

17:51 And people who have this sense of community, being humble, to be able to work together. And this is where I see in Woosuk, seeing Kai Chao Sing, as well as Roger, that we have this team where everybody is extremely humble, extremely intelligent, but also have this ability to work together to really work out ambitious hard problem. Inference after all is a very very hard problem to solve. So, we're building this team together and realizing that to in order to solve the hard problem, you need a team of folks who understand everything inside out, be a little bit opinionated, but also be nice to each other, be able to figure out we're all in this for the technically deep problems that's decades to come. So, AI is evolving at breakneck speed. How are you making sense of that and how are you keeping pace with it in the context of the way you're designing the company? AI is changing indeed and inference is changing drastically. From my point of view, the only competition that we have is vertically integrated specialized inference engine that is in-house and proprietary for proprietary model.

19:00 What we are building here is an entirely open ecosystem and a horizontal stack that covers everything and invite everybody else to contribute. And from our point of view, our real users, our end users, are all users of vLLM. So, that means regardless of anyone, as long as they are using vLLM, deploying vLLM, and need our help within their services and within their hardware, within their model, we're there to serve them and want to help them to make them successful so that we can all together make vLLM the best inference engine in the world. As we all know, this was a this the process of company formation here, you know, was a long and interesting journey. And the point at which you did decide to fundraise, I'd love to learn why you chose to work with Lightspeed. What was it about Lightspeed that that appealed to you? Yeah, good question. Lightspeed has been supporting the lab, Sky Computing Lab, as well as working with the graduate students of our lab over time on different things, right?

19:59 So, in the beginning, it's not even about commercialization at all. James, when we first discussed and talked about, it's not about like, how do you actually monetize? Like, we didn't even talk about that. It's all about here is an open-source project. Here is my user so far. How do I grow the open-source project? How do I know who's using it, who's not using it? Why are people not using it? We talk about product, we talk about the market, we talk about the gap between research and the real world. A lot of the conversation really is helpful to graduate students and students working on open-source project to really materialize and bridge the gap and their dreams. So, this is also why I believe LightSpeed and all the partners together are really there to help to bring technologies to the world, help us to do so. And that's why we chose LightSpeed, too. We feel lucky to be on this journey with you and on that note, we're here today to to talk about a financing round. And obviously, with that, unveiling the company. What is all this capital that you're announcing you raised today enable you guys to do?

21:03 So, the capital we raised today will be used for talent and as well as compute needed for developing the cutting-edge method and research around to bring the inference to the next level. Because inference is no longer a single chip activity. Inference no longer run there, it runs on a big cluster, it requires constant upkeep, constant maintenance, constant benchmarking, understanding. And then with new model being released, we need a big team to support different kinds of model, different kinds of soft hardware. And then really go in and deep dive into every single user's custom use cases and workload. I I fundamentally believe this world is so vast, there are so many ways to use a model on different chips that we need the capital to enable the entire team to build VL to really across the whole world to become the world's best inference engine. And we And we talk about inference really shifting kind of in lockstep with the way models are being used and applications are wielding them.

22:05 Today, obviously, where where you hear a lot about these longer horizon agentic tasks. Like, talk to us a little bit about how you think about the way inference is is going to change as a result of this this new paradigm. Yeah, so, if you look back, inference is just about chats and then completions. So, one input, one output. But nowadays, inference is becoming agentic, stateful. And this where we also see the transition from what called an inference engine to inference system to inference platform. That means going from a single input-output kind of a system to a complicated distributed system to the entire platform where you not only need abstraction on top of the on top of the assorted computing devices inference engine, but also state management environments and everything you need to enable a agents agent working like 8 hours, 16 hours, 24 hours long horizon task. And fundamentally, this cut across the entire model to hardware to sort of the infrastructure co-design. So, we need biggest GPU out there, right? We need GPUs that are even more interconnected. We need GPUs that are enabled us to share multiple models, share multiple inference stream together. And then be able to implement all sorts of transition to support long-running state as well as supporting the complexity of the request. If the dividing line between training time and test time sort of continues to dissipate somewhat, what does that imply about how inference systems should change? So, inference system will then really becomes a core part of training, first of all.

23:47 As more rollouts needed for the RL process, as pre-training essentially becoming a known quantity of a capital investment, the biggest scaling paradigm on the RL front, on the post-training front, and then in the finally, when you start interacting with the real world, this where inference absolutely dominates. That's why I've been reading all the news about data center buildout and new capacity being added. The way I'm reading it is all of this will become inference compute. The cluster are being built for training, but six months later, the cluster will be used for entirely for inference.

24:23 And the new cluster as they will build, new model will be trained. Sure, great. But then, you know what? Very soon, inference will start eat up the capacity bit by bit and then be able to really consume all the newly provisioned energy to be able to support all of us today. So, I mean, this was awesome. We're so grateful to to be your partner and excited to be on this journey with you and your co-founders. Is there anything you want to leave our viewers with? We could talk about hiring, we could talk about anything else we haven't hit on.

24:54 What what what should our viewers know that they haven't heard yet about Infract and the project? Yeah, so, it's really much the co-founder have this vision of building this very technically deep, very interesting system. And fundamentally, we are seeing a paradigm shift. We are seeing a paradigm shift from building vertical system, like model, hardware, just build that for specific model. Like, from what I've been hearing before, for example, OpenAI, there's inference engine for Sora, there's another inference engine for new version of Sora, and one inference engine for each model even.

25:28 But then, for us, we're building one inference engine for all the current future model, for all the current future hardware, and for all the current future applications. This is a new operating system, this is a new kinds of database, is a new kinds of system paradigm discipline for current and future member of Infract to work on. And so, we are quite excited about the journey ahead and how deep and how technically deep it is and how impactful it can be. Well said. Congrats on launching the company today. Thank you so much.

25:57 >> Let's get back to work. Yeah, let's go. >>

Summary

The discussion centers around the evolution of inference in AI, highlighting the significance of vLLM, an open-source inference engine developed by Inflection. As data centers expand, the focus is shifting from training to inference, with vLLM positioned as a key player in optimizing model execution across various hardware and applications.

- Inference is becoming the primary value driver in AI, with vLLM at the center of this shift.
- vLLM was co-created at UC Berkeley and has rapidly gained traction as a standard for AI infrastructure.
- The project emphasizes maximum efficiency, throughput, and speed in processing AI models.
- Paged attention is a core technique in vLLM that optimizes handling non-uniform input and output in language models.
- The open-source nature of vLLM has fostered a vibrant community of contributors, including model and hardware developers.
- Engagement with hardware vendors like Nvidia and AMD is crucial for ensuring compatibility and performance across different chips.
- The future of inference is evolving towards stateful, agentic tasks, requiring a shift from simple input-output systems to complex platforms.
- Inflection aims to create a universal inference engine that supports all current and future models and applications, marking a significant paradigm shift in AI infrastructure.

Questions Answered

What is the significance of vLLM in the current AI landscape?

vLLM is positioned as a leading inference engine that supports a wide range of AI applications and hardware. It is built to serve the needs of users deploying virtual machines (VMs) and aims to become the best inference engine globally.

Why is inference critical in the AI model development process?

Inference is where the real value of AI models is realized, as it allows for the application of intelligence in practical scenarios. Efficiency, speed, and throughput are essential for maximizing the benefits of AI.

How has the vLLM community evolved and what role do hardware vendors play?

The vLLM community has grown with contributions from companies building models and optimizing hardware. This collaborative environment fosters innovation and allows for shared benefits among participants.

What is necessary to create a comprehensive inference platform?

To build a successful inference platform, additional components must be integrated with vLLM to achieve significant efficiency gains. The focus is on open-source principles and a culture of innovation.

What will the recent funding enable for the company?

The newly raised capital will support talent acquisition and computing resources necessary for advancing inference technologies. This funding is crucial for adapting to the evolving landscape of AI applications.

© transcribe · For agents Built with care and craft by Gokul Rajaram