Great Infra for Background Agents | Sail

0:00 Everyone, excited to be here. as Brian mentioned, I started this company called Sale Research to solve this problem of batch inference. so let's talk about what we're building, why we're doing it, and how we're going to achieve greater better economics for everyone. So, as a quick agenda, I'll start by just describing my view of inference, what, you know, what I've seen in the industry for the last 10 years, evolving from early language models to modern agents. we'll talk about what's going on with autonomous agents today and where they're kind of getting stuck in my perception.

0:30 we'll talk about how we choose to optimize for throughput or latency, and it is a binary choice. And finally, we'll think about the future of where abundant inference could be going. So, we'll start with the history. I think that you can look at the timeline of language model development in three distinct phases, or here's the at least the framing that I like. we saw the beginning of the LM era with something that was inherently human interactive. The first goal of AI, of language models was to engage humans in conversation. and the point of that was just to show that token prediction could go somewhere, that we were having something interesting coming out of these streams of next token predictions.

1:11 They weren't super useful, but they were highly engaging. We saw some improvement once we got to the point of understanding how tokens could actually produce better results if you could spend more on them. we started to see this test time scaling paradigm pick up pick up in 2024. again from OpenAI, the idea that reasoning models were going to give us a new axis in which to develop better intelligence. But the one I'm most excited about, the biggest change in inference in the last 3 years, I would say, is this move to actually letting the agent take or letting a language model take actions, something that we call an agent now, but what I define or many people define as a reasoning model that's given a loop to call tools in so that it can take action as it pleases.

1:55 All three of these phases of language model development have been about inference. But, the actual shape of the workload here is dramatically different between tokens that we use for chatbots to the early generation of reasoning models and to what we have today, highly autonomous agents. So, let's start by thinking about the dichotomy between what chatbots were and what agents are now. Ben Thompson at Stratechery actually just rated wrote a really great piece talking about the shift towards agentic inference. And I think if you want to talk about We all agree that agentic inference is the era we're living in now, but I think it's useful to define what we came from. I've used the term chatbots, but Ben came up with this new term called answer inference, which I like quite a bit. Essentially, you're a human waiting for an answer and you don't like waiting. So, generally speaking, answer inference, where you're a human in the loop waiting for a chatbot to get back to you with an answer, needs to be fast.

2:49 there's a balanced ratio between the user's effort and the machine's effort, and the user's typically the limiter in terms of how much work can be done or how fast the progress can be made. but, the most interesting thing about answer inference or what kind of keeps it in the rearview mirror for me is that the human in the loop ends up being the strong bound on how much we can how much how many tokens we can consume here.

3:14 if you're the bottleneck on how many tokens can flow through your system because you're typing in all the inputs and reading all the outputs, I don't think we're actually going to see dramatic scaling from where we are today. ChatGPT is already getting close to a billion weekly active users. We're getting to saturation with answer inference. But, we're excited about agentic inference now, where the human is getting more and more decoupled from that loop, and we're giving the agent more flexibility to run as long as it wants to for whatever task is at hand.

3:44 Now, if you project the work that we send to AI models over time, here I'm plotting kind of this abstract sense of productivity on the Y axis and on the X axis the amount of time that each agent is running for. I would say I would argue that we've been in this V-shaped curve. roughly where we started was on the left where the agent would respond to you in just seconds. you wouldn't wait very long for an answer and that was somewhat useful.

4:10 we got to the point where the agents were running for a little bit longer. We would give the agent more time to think or give it more time to call tools and they were still quite useful but it started to get annoying to basically sit there and wait for the agent to to finish speaking to you. You would be blocked waiting for the agent to come back to you with an answer whether you're in cloud code or cursor or even just chat GPT's window.

4:31 And now we're actually in this really tough spot in this in the middle this nadir of the valley where the agents want to run for longer. We're getting to the point where they can run for 5 or 10 minutes and there's really not a whole lot we can do but wait for the agent. You can try to pay more to get it to happen faster. You can pull the curve back towards the left here a little bit but it gets expensive and ultimately there's this trade-off between letting the agent run for a while versus having it do ambitious work.

4:58 This is the most painful part of the curve to be in but it's where we are today. What we have to have hope for is in the next coming months we will see the agents be able to take on we can trust them with more responsibility and more scope and we hope that they'll actually start running for longer and longer. And the reason I think that that will increase productivity instead of continuing this downward trend here is that as we delegate more large chunks of work to the agent we'll have less time spent for for the human in the loop waiting at the computer waiting for an agent responds. You will kick off an hour long task and go get coffee go into a meeting. You'll kick off a 4-hour long task and maybe just clock out for the day.

5:37 You'll kick off a 24-hour task and now you're starting to treat the agent more like a human employee. This is now a human time scale. You don't expect work to be done from human employees on like a 5 or 10-minute basis. You give it a day, you give it a week, and you come back when the task is done. And that to me is the promised future of long-term productivity for agents. But, we're here. I want to be optimistic. I want to think about what's going to happen to get us to that long horizon.

6:06 in that world where we are working on long horizon tasks, we need to think about throughput more than anything else. How do we choose throughput when we're building an inference stack? This image is a little bit small. Let me try to zoom in a little bit more. Okay, so the inference stack, in my view, has, let's call it six layers, from the physical infrastructure at the very bottom, data centers, chips, power, all the way to the API that we all consume at the top.

6:33 at every single one of these layers, you can choose between throughput or latency. And unfortunately, it is a binary. You don't get to choose both. This is one of the oldest tradeoffs in all of computer science and really every field. You must choose to prioritize one over the other. So, let's actually stop from start from the top down. I think that's an easier way to get into this. for the APIs we started with, I would argue that most companies, like OpenAI and Anthropic, started off on the latency side of the spectrum because we all started with these answer inference chatbots.

7:03 >> >> And so, if you just look down the right side of the of the lane here, I'll explain a few things that are maybe new here. But, we started with APIs that were streaming responses back to users in real time. we've put a lot of effort into routing requests and setting up data centers across the globe so that we can be close to users and get data get responses to them as quickly as possible as the network latency is an important thing.

7:25 And we've built all of our GPU systems, our inference engines, like VL M SG Lang, around all these techniques that optimize for pushing tokens out as quickly as possible so that you never keep a user waiting. and you might have heard this before, but a lot of the work we do in modern inference is about dealing with the fact that GPUs are actually memory bound when they do inference. We are generating tokens one at a time thanks to auto regressive language models, and we end up just being stuck like moving small amounts of work on and off the GPU essentially. and we end up actually using the GPU in the bandwidth bound regime.

8:03 and so we built kernels silicon like as in programs that will be on the GPU. We built the silicon around this. We have a very expensive memory like HBM on the latest Nvidia chips, all optimizing for memory bandwidth over and that that turns out to be quite a driver of the cost as you've seen from the recent memory shortages. and then most interestingly to me, data center design tells the story most cleanly. Data center design has always followed the points of presence on the global on the US network. We follow the fiber lines in terms of where we think we can build data centers.

8:35 And that has really constrained the sites that we can build data centers, and you might have heard that it's hard to find capacity to put compute in place. this is why. We have have a had a rather narrow view of where we can afford to put a large computing cluster. all because we've been optimizing for latency. So, if you flip over to the left side where we think we need to go, let's again go through from the top to the bottom what we would change about the stack to be optimized for more throughput instead of latency.

9:04 The first thing is actually something that OpenAI has contributed. We have a new API that's being used for agents that's being promoted by OpenAI called the responses API. One of the most interesting things it does is it focuses on asyncronicity, meaning you send in a response and you don't get a response back immediately. You don't hold the HTTP connection open. You just wait for OpenAI to tell you maybe by calling a webhook, or you pull. You wait for the service to tell you when the response is done.

9:30 This is about setting expectations. It's not going to be done in seconds, it's going to be done in minutes and maybe even hours eventually. But, this is the first thing we had to get right. Opening I had to lay the foundation with the responses API. The second layer, where we actually think about routing these requests towards, I no longer think we should be following the users' location as much. We should basically be thinking about where we have compute capacity at that moment in time. We should be using the night time cycles in Asia to cover the daytime cycles in in the Americas or in Europe.

10:00 We actually have round the globe inference cluster deployments, so we might as well use just cycles to our advantage by moving inference around the world to follow usage patterns. And here are the layers, the inference engine, the kernels and silicon. These are the bread and butter for our company sale, where we invest the most work into rebuilding the stack around throughput. Inference engines have to focus more on throughput than anything else now. And that means choosing parallelism schemes, it means expecting to run very large batch sizes.

10:34 It means choosing to sacrifice latency, tokens per second will go down while optimizing for peak GPU utilization. And we'll talk a bit more about what that means for silicon design as well. And then finally for data centers, that's one of the most asymmetric bets we would take is that we don't think we should follow the internet connectivity. We think the internet can actually be less of a factor for inference than than you might think. It's not that many bytes to move tokens across the wire.

11:03 What is hard is to find power capacity, finding clean abundant power. And so we think that the generate data centers of the future will actually follow abundant power more than anything else. And we'll talk more about that as well. But we make all these choices. We've pushed hard towards throughput. We can't get latency, but what we can get is much better costs. So here we ran some reference workloads. We'll go into the benchmark that we ran to can this later, but roughly speaking for a given model, sale by making choices across the stack to optimize for throughput is able to get something like five to six times better than even the best other providers who are optimizing for latency.

11:45 and this is the real reason we do what we do. We want to make tokens actually abundant so that we can do what's next. So the reason we choose to focus on token abundance is that another factor is that we believe in parallel intelligence more than we do serial intelligence. What that means to me is instead of focusing on just spitting out more tokens in a single line like we do with current reasoning models, we actually think that more and more we're going to see multiple agents working on a problem in collaboration.

12:17 having more shots on goal gives you a better chance of getting the answer right even with the intelligence level that we seem to have today. It is one of the most interesting new ways to scale agent performance. and in particular, we think there are a few problems that are particularly well suited to parallel scaling instead of serial scaling. My favorite is deep research. Most of the problems in deep research, which involve looking at a hundred or a thousand documents to find the most relevant information for a query, most of them are context limited.

12:45 It doesn't take a genius to find out which document is relevant to the query, but it does take a very tenacious agent to actually be able to look through thousands of documents to have a chance of finding a needle in a haystack. Similarly, code review is becoming a fast-growing use case for this parallel intelligence model. There's no need to have just one or two agents looking at a code base. It's a static problem and you can spend as many tokens as you're willing to basically pay for looking for bugs in code.

13:12 And most broadly speaking, we think that giving agents more chances to try roll out a full idea, see where it takes them, and then post factor decide whether it was a good path or not. That we think is the way that we're going to see agent intelligence improve over the next year, rather than praying for simple straight-shot IQ improvements in a single line. So, for deep research, we've actually started with that problem as the most exciting use case for for sale, our company's inference API. We made tokens abundant, so we thought we would prove how well we could do by trying to win a benchmark that does seem to be context limited.

13:49 We came up with Browse Comp Plus, which is one of the best benchmarks out there for deep research capabilities. The task asks us to look through a corpus of about 100,000 documents, about 1,000 queries on against these documents, and figure out basically correct answers to each of the queries. it's a lot of documents to look through, and you have to use a hybrid retrieval system. You have to first filter down the documents in the corpus down to what you think is most relevant, and then have an agent go look through each one, read everything, and figure out if there's an answer there or not.

14:25 It's the perfect needle in a haystack problem, and one that we think agents are really well suited to do as long as you're willing to spend more tokens on the inference. we're proud to have the best open-source model score on the leaderboard as of this month, and we think that that's actually the pattern that we're going to keep seeing. models for retrieval and search are actually getting to a point where their intelligence is saturated. The challenge is mostly about scale and getting to give them more chances to look for documents. Look through the bigger haystack, better chance of finding the needle.

14:59 Once we've done that, we started to look elsewhere in the agent stack. once we have abundant tokens, we think about essentially what does What else does an agent need to be very ambitious? And one pattern that we've seen recently is the need for agents to have computers. We know that computers are critical for the tool use paradigm of agents. We know that we're We're need a lot more computers than we what we have today. I don't think the agents of the future are going to run on my laptop or on my Mac mini.

15:26 we think that cloud agents are going to scale to maybe 100 x more demand than what we see today. We're in the early innings. And so we wanted to test out or we wanted to build infrastructure that supports abundant inference just as much as we need abundant sandboxes. We built a sandbox API in private beta that we're focusing on basically scaling to much more many more independent sandboxes that can snapshot and resume very quickly. And most critically, we built a sandbox that is able to not bill you for any time you're not actually doing work.

16:01 A sandbox that is able to go to sleep when it's waiting on inference because we know it's going to run for hours on end. And to test this out, we set up a software development task where we asked the agent to build a wire compatible Redis clone. it took 24 hours. It used a swarm of agents. And they consumed a lot of computing resources in aggregate. However, they were not billed for all the time that they spent waiting on inference.

16:26 and that's about a 2 x improvement or 40% reduction in the theoretical cost of that of that task. The reason we were able to do that is well, we actually if we control both the sandbox compute as well as the GPU inference compute, we actually are able to know at any given point in time is the agent thinking or is it acting? And one interesting property of agents is that they don't do both at the same time.

16:49 They're either on the sandbox or they're on the GPU doing inference. And if you can see both sides of the workload, well, it's kind of a scheduler's dream. We're able to know exactly when to schedule work on the GPU versus when to pause it, page the KB cache out to disk, and vice versa on the CPU side. We're able to move memory in and out of the system in order to actually meet the needs of what agents are currently doing what phase.

17:14 We think that this pattern is going to keep coming up where people who are able to make more use of the same fixed resources that we have are going to see better scaling and better results from the finite compute that we're all going to be bidding more for in the next couple of years. All right. And then to talk briefly about what this means for compute, I think that the shape of inference demand and how it grows is going to have profound implications for how we think about Nvidia chips and alternative chips in the future. At a high level, I'm very bullish on alternative compute at this point. I think that there's actually no secret in how you build a great AI chip, especially for inference. It has focused the workload so much and there's going to be lots of competition to Nvidia. But not in the traditional sense. Nvidia will still be a great company because they've invested very heavily in low latency optimization.

18:03 They are still untouchable for using multiple GPUs together for a training job, still great for getting the fastest tokens out in the world. Good good reason for them to buy Groq in the last year to focus on low latency. But everyone else is going to come for the throughput regime. It is easier than ever to build an AI chip. You have 10 different competitors that all have the same core idea. AMD, I think, is one of the leading ones and one that we're very excited to work with over the next couple of years. They're using commodity technologies wherever possible, Ethernet instead of InfiniBand.

18:36 They use chiplets. They've been an early proponent of basically packing compute as efficiently and densely as possible on the manufacturing processes that we have. And they're able to pump out a lot more compute per dollar than Nvidia can. I want to talk a little bit about also the broader trend on where silicon is going with you know, what we think of Moore's Law in this modern age. Basically, we're seeing that it's been widely reported that Moore's Law is not nearly as fast as it used to be. We're no longer getting two-year node transitions. It's more like three. And the step change we're seeing between one node to the next is not nearly as dramatic as it used to be. We expected, you know, 1.5x or 2x before. Now we're lucky to get 20% improvements in performance per watt.

19:22 And all that said, the capacity of these new fabs is also getting scarcer than ever. It's really hard to build new fabs. Everybody's talked about that. it's quite hard to imagine how we're going to dramatically increase the supply of net new fabs in the next few years. So, I wouldn't expect TSMC to save us from our compute crunch that's looming. What I do think is going to be interesting in the next few years though is that because Moore's law has been so slow in the last couple of years, we might actually see a return to older fab technologies. the reason I think this is the primary cost of the data center today is not actually power yet. It is one of the constraints, but it's not the cost that actually drives the data center or the compute scarcity.

20:03 What's scarce is the chips coming out of these fabs. And if you're able to make that much more abundant by using older process nodes that are stable in friendly countries and so on, we might be able to dramatically increase the supply of chips, maybe not frontier chips, but some chips, and take only a 50% or so haircut on peak performance. And for inference, that's a very good trade because it allows us to use much more abundant power, for example, solar and wind out in the wide fields of the US or even possibly space data centers. I think these will be great fits for abundant silicon and scarce power, which is where I think ultimately our limiters are going to be in the next 5 years.

20:48 And then finally, I wanted to just close with the small recap on kind of where we are with inference and where I think we are kind of converging to. Essentially, we think that the future of tokens is to be produced by machines for machines. We're not going to see a lot more humans in the loop as a percentage of the total token in the world. we will continue to see humans in the loop, but only at the result level.

21:14 Tokens are going to be a thing that only machines have to think about. In response to that, we will see a restructuring of the whole stack, like we mentioned earlier, from the data center level all the way to the API level, around high-throughput inference over anything else. Efficiency and scale are the most important things in this world, not speed alone. And finally, we of course having a company to build in this space, we think that vertically integrating across all these layers is going to be an extremely powerful advantage for people who want to build the frontier of inference systems. We want to squeeze every joule out of every power source that we have and put that to work on all the silicon we have accessible to us.

21:51 that is the plan for sale, and that is the plan for us to deliver cheap tokens to the whole world. Thank you for for coming to the talk. >>