Section Insights
Introduction to the Largest Chip
What is significant about the largest chip built in the history of computing?
The largest chip, built by Cerebras, is 58 times larger than a GPU and is designed for faster AI processing. Despite its groundbreaking capabilities, it initially faced skepticism and low market interest.
- Larger chips process information more quickly, making them ideal for AI.
- Cerebras faced initial rejection despite creating a revolutionary chip.
- The transition from GPU to their chip is simplified for users.
Cloud Computing and AI Demand
How does cloud computing relate to AI processing demands?
Most real compute tasks for AI will be handled in data centers rather than on local devices. Cerebras focuses on data center compute to meet the high demand for AI processing.
- AI processing will predominantly occur in data centers.
- Local devices will handle smaller tasks, while larger computations will rely on cloud infrastructure.
- Cerebras aims to optimize data center compute for AI applications.
Challenges in Chip Development
What challenges did Cerebras face in developing their chip?
Cerebras recognized that simply improving GPU technology wouldn't suffice against established competitors like Nvidia. They aimed to create a fundamentally different architecture, which led to a challenging development period.
- Cerebras aimed to solve a complex engineering problem rather than just incrementally improve existing technology.
- The company faced skepticism and had to navigate a difficult market environment.
- Building a new architecture required significant investment and perseverance.
Speed and Efficiency in AI Processing
Why is speed crucial in AI model processing?
The speed of moving weights from memory to compute is critical in generating AI responses. Cerebras's architecture allows for significantly faster processing compared to traditional GPUs, enhancing overall efficiency.
- Moving weights for AI processing is a resource-intensive task that can be optimized.
- Cerebras's chip architecture enables much faster data handling than conventional GPUs.
- Efficiency in processing directly impacts the performance of AI models.
Flexibility in AI Solutions
How does flexibility play a role in AI chip and service offerings?
Cerebras has adapted to provide both chip solutions and data center services to meet diverse customer needs, focusing on delivering fast AI processing tokens efficiently.
- Flexibility in service offerings is crucial for meeting varied customer demands.
- Cerebras provides on-demand services to facilitate AI processing.
- The company aims to streamline the delivery of AI solutions through its infrastructure.
Transcript
0:00 This is the largest chip built in the history of the computer industry. It's 58 times larger than a GPU. And for AI, bigger chips process information more quickly, and therefore you get answers in less time. For AI work, big chips are undoubtedly the best way to go. There's no mode and inference. It takes you eight keystrokes to move from a GPU to us in the cloud. We solved a problem that nobody in the history of computing solved, and we delivered it in 2020, >> >> and nobody cared. Nobody cared. Nobody bought it, and nobody cared. Everybody said we were crazy, it would never work.
0:39 So then we built the next one. >> Hi Emmett here. Welcome to the Mad Podcast. My guest today is Andrew Feldman, co-founder and CEO of Cerebras, the company that built the largest chip in the history of computing and just pulled off the biggest semiconductor IPO of all time. Andrew has been everywhere talking about the headlines, the $20 plus OpenAI deal, the IPO, but this conversation is a bit different. We started from what is a wafer and built up step by step why GPUs struggle with fast inference, the three shortages nobody talks about, the decade in the desert when nobody wanted his chip, and why Andrew believes that CUDA is no longer a mode for Nvidia. If you want to actually understand the current chip landscape and how AI inference works at the silicon level, this episode is for you.
1:25 Please enjoy this fantastic conversation with Andrew Feldman. >> I feel like a a fun place to start would be to talk about speed. So how has speed become the dominant conversation for AI today? >> Yeah, what happened, I think, was for a long time AI was sort of a novelty. All right, it was like a parlor trick. It was cool, but not useful. And what happened somewhere around the middle of 2025 was the AI got smart enough such that people began to use it. And we remember we we we make AI with training, but we use it with inference.
2:06 And suddenly people wanted to use it. And the minute you want to use it, the minute it's productive, right? Speed matters, right? Fast tokens are more productive. And so the conversation moved from everything else to how do we make our inference faster? How do we deliver tokens more quickly because those are more productive tokens? We get more done in less time and therefore it's more valuable. Right? And what does speed mean? Is that is that a question of token speed? Is that completion of the task? What's the right metric? So the right metric is tokens per second per user. That that's how fast you get the first token all the way through the last token in in your in your response.
2:53 And it's true for queries from from chat, but it's also true for for agentic floats, right? If there sort of multi-cycle turns, waiting is amplified. And so what you want is blisteringly fast responses so that the the AI feels like it's in real time. You can engage with it. Okay? Okay. So it's the broadband moment for Australia. I say. Yes. I think that's right. And I I think that's a very good analogy. I I think if you think of something like like Netflix, right? When the internet was slow, right? Netflix delivered DVDs in envelopes. You would get a a DVD in an envelope.
3:36 And when the internet became fast, they didn't get more efficient at delivering DVDs in envelopes. They became a movie studio. And right? The speed enabled them to become something completely different. And that's what speed does in general and in particular for for AI. It it opens up a whole new domain that allows you to use the AI differently. You will stay longer. You will come more often and you work on harder problems. >> Yeah. So, it's literally a question of of the UX, right? That's just nobody wants to wait a few seconds.
4:11 >> Yeah, that's right. I mean, how how big is the market for slow search? How big is the market for dial-up? Is zero. How big is How long will you wait for a website to resolve? Will you wait 8 seconds? Nobody waits. See? And so, it's the exact same with AI. >> Yep. So, no more people waiting with their laptops open and while the agent >> That's right. While it's running and running and running. I I I think that is not what what people want.
4:36 >> Okay. Great. Wonderful. So, I'd love to talk about the landscape of the chip industry right now. Just help people visualize and part where where where you guys are. So, there used to be basically this concept of like one chip to do it all. and obviously this is evolving dramatically. this you guys this this Groq this there are GPUs, there are and people have heard of Tranium, people have heard of TPUs. So, help us compare and contrast like who does what for what.
5:16 >> Well, I I I think there used to be one chip to do it all. It was called a CPU. Yeah, full through, right? And there emerged a co-processor to do discrete graphics. And as the AI workload became interesting, we focused chips more and more on that particular workload. And today several companies make traditional GPUs. So, Nvidia and AMD. They make very standard GPUs.
5:46 there are a group of companies who the hyperscalers make make some of their own parts. So, the TPU >> So, to me it's Google. >> TPU is Google and Trainium is is AWS. >> Yep. >> and then there were group like Cerebras. And we were among the pioneers to to to build a part from scratch optimized for AI and nothing else. And that we were inside of hyperscalers. We weren't optimized for hyperscalers problem or for one labs problem.
6:23 We we were building a a chip for a collection of AI problems. And all our thinking was around how to accelerate AI. >> And that's sort of the landscape to this day. >> Yeah. >> There's even more specialized ones, right? Like Edge develops special transformers. I think those are called ASICs as well. Can you maybe define what that term means? So, >> An ASIC is an application-specific integrated circuit. And and it's it's a word that that now has a wide range of of meaning.
7:01 It it means that you have made a series of choices away from general you towards a narrower class of problem-solving. And that you've made some decisions in your architecture that make it much better at some things and much much worse at others. Right? And and that's choices that are made across the spectrum. So, the TPU has made some choices like that. It can't do graphics.
7:31 It is very good matrix multiply. It's not very good at a collection of other things that we do in in mathematics. same for the GPU. We've all made different choices. We've all made different choices. Ours right now in production there are, Iridium and Andy. There are a collection. There's a TPU for Google, Tradiums, just coming up as a Maya part is a part from, Microsoft, Cerebrus. And there was one other, Groq that got acquired by by Nvidia.
8:10 >> Yeah. And to the general chip versus specialized chip, it was actually interesting because I you guys and your CEO celebrated when Groq was, acquired. Was that So, what was that? Was it a recognition by Nvidia that your vision was right all along? >> Yeah. I I think, one of the things Nvidia's sort of most durable moats was the perception that the GPU could do everything. You know, it was all you needed for AI.
8:41 And the acquisition of Groq for $20 billion and the structure and the speed with which they chose to do it made clear to everyone that that wasn't true. That, the GPU architecture couldn't do could not do fast at trends and that this market was large and growing quickly. And we were the fastest at it and the largest. And, you know, our sales were more than 10 times Groq's and they paid $20 billion for the number two player.
9:15 So, that was a good day. >> Indeed. >> That was a good day. >> And, the recent announcements, of Halapenio between Open AI to the very large customer of yours and Broadcom, where does it fit in that picture? Is that one of those highly specialized ASICS types, right? Groq is this for inference? >> That's right. Halapenio is, a long time coming. I'm a it was announced with Broadcom before Open AI did the deal with us.
9:49 We have but we did a huge deal. This is probably the largest deals in Silicon Valley history, north of $20 billion. I I think today have a a yawning need for silicon. And we one of the things that Open AI has been, I think, the best have a in the industry at is looking at an exponential curve adoption rate of growth and not being afraid of what it says, right? Others have had to go out and strike really bad deals to get capacity.
10:28 Where Open AI saw this coming, they struck big deals for memory, for compute with us, with others. they've really been sort of visionary in understanding the what it means to extrapolate from an exponential curve. And that curve is is AI usage. Right? It is growing so unbelievably fast. The whole industry is chasing demand, right? Right? We usually well, often it's the other way. Often people are building it hoping it will come.
11:01 And in our case, all of us are are are chasing what people already wanted to. Let alone what they might do in the future. >> I think what I mean in that I mean let me cuz this such an interesting topic. to to to finish on GPUs versus inference versus ASICs, is that the permanent situation from your perspective that we're going to be in this forever multi-silicon tide environment? >> Yeah, multi-silicon environment is a healthy ecosystem, right? I I don't think anybody would say that the x86 environment was was a healthy place.
11:38 There were for 20 years there were two players. And if you notice when a new workload came around, cell phone workload, that was very similar but required low power and battery and both lost. Right? Intel got zero share, AMD got zero share, and a company nobody previously heard of became the largest seller of computer in the world and that's ARM. A A And that healthy ecosystems have lots of different ways to to solve problems.
12:10 >> Where where does China fall in all of this? So there's Huawei that send chips for deep sea. Can this emergence of full stack Chinese AI factory for lack of a better term? Is that Is Is Is Is Is Is anybody happening or is that overstated? I don't know that >> device is real. >> No, I I think they are an industrial adversary. >> They I'd say I'm a They have made really interesting investments. They made investments in power and the grid, which is a real weakness in the US.
12:45 I'm sure in France you you have nuclear, which sort of turns out to be pretty cheap you know power and pretty clean. but in China they made huge investments in their grid. they are behind in chips. but their approach was the next level was open source models where they're producing some extraordinary models. Not as good as as as GPT or Anthropic or Google's Gemini, but very good. A And I I think they don't have the the same chip strikes that the US does, but they've leadership in other domains and potentially in power, which is what we need for data centers.
13:32 >> Okay. Where where does China fall in sort of race strategy? >> We don't sell fractions. >> for geopolitical reasons, for business reasons? >> Well, we regulatory reasons. >> For regulatory reasons and and and for some geopolitical reasons. >> Okay. is there a role for local AI and local chips? Nvidia had some announcement around just building chips for Windows computers. Is it for inference? Is it something that you guys think should be part of the multi-silicon ecosystem?
14:09 >> Yeah, I I think if you look at the way the ecosystem for apps and cloud emerged, everything you can do, you should do on your phone or your laptop. I think but the the the ability to get real processing power to a phone or to a laptop is constrained because they're generally working off a battery. And so you you you want to do as much as you can as as close to the data as you can.
14:38 But the truth is is in most situations for real compute, you have to go to the cloud to the data center. Yeah. And that's exactly the way it's going to be with AI. We're going to do a lot of work off on the cell phones, on the laptop, but for the big work, you're going to go to to the data center. And that's where our focus is. Our focus is is data center compute free. >> Okay. Right.
15:02 So as you mentioned the the how powerful the demand was and the demand at a you know, at all production. So just to ask the inevitable question around market equilibrium, bubble or not. And so like the most recent I think we all saw was that there was a little bit of a flash crash earlier in June where 1.5 trillion in market value was lost and and India had lost 300 billion that day. Now, that was corrected within, you know, 48 hours.
15:33 So, the market seems very jittery around the that that that general concept. Maybe that's just what I put in words in your guys' own just based on what you said. That sounds like you're you're really going to play on the other side of that. So, great concern. >> I I I I I take that that if you spend a lot of time staring at the at the public markets and watching their ups and downs every day, you're you're watching the wrong thing. I I I think that right the wisdom I received from the great thinkers of public market investing is that in the short terms, the public markets are are voting mechanisms. Who's most popular?
16:19 But in the long term, they're weighing mechanisms. Who's created the most value, the most weight? And so, what we can do as a newly public company is is we cannot look at the day-to-day fluctuations and we can focus on I just building extraordinary technology, winning new customers, making our existing customers happy, and ensuring they buy more. being better every week. And and so, I I think that that the day-to-day fluctuations are they're are out of our hands arms and that that's about the voting.
16:53 >> Yeah. And let me just to play in devil's advocate on the demand. it seems like a lot of the demand comes from the big labs, which you know, themselves are financed by venture capital, private equity, hedge funds, whatever you call it, and so on. You know, and sovereign investors. is there is there any concern that, you know, demand for chips comes from the lands which are maybe artificially financed and that the you know, if you had a few circle of deals here and there, do you sense any fragility?
17:24 >> I I That's not exactly our experience. I mean, obviously we have enormous demand from people again. But we have huge, you know, dozens of other customers who who are trying to place in very big orders with us. And But historically bubbles were when supply got out ahead of demand, right? When in the '90s we built out telco infrastructure. Right, we built out fiber years before it it was going to be used and it took 6 or 8 years and it all got used.
18:06 But it it was a sort of if you build it, they will come mentality. Whereas what's different about AI right now is we're all trying to catch up. we're trying to build data centers faster. We're trying to increase our our demand, our our our supply chains for demand that's already here today. And only a very small portion of the the world are are using AI anywhere close to its potential. And we're already close sort of overwhelmed with compute.
18:36 We're overwhelmed with the demand for for memory, which is a real weakness in the GPUs. It's not a problem we face. the ecosystem, there are constraints left and right and that doesn't feel like a bubble. >> Did you want to actually take a multi-take on that because that's So interesting what you write is it seems like the bottleneck keeps moving because So what's the current problem of memory shortage that people have heard of?
19:07 >> So there are three major bottlenecks right now. I want to the first is all GPUs and all ASICs except us use a type of memory called DRAM and a particular flavor of that memory called HBM. And that memory is made by three companies in the world. >> SK Hynix, Samsung, and Micron.
19:38 >> Three companies. And they're sold out. That's number one problem. We don't use it. >> And it's sold out as in like the only way to as all things hardware, the only way to increase the supply would be to build new factories? >> more factories. And all of them and so you have this huge problem for GPUs and even now for CPUs and which you can't get this memory. Because of our architectural choices, we we don't use it.
20:10 The second constraint was a process at TSMC. Process was called CoWoS. And this used silicon as a motherboard. And this is what Nvidia and AMD and others use as a motherboard so they put their chips and the memory chips on it. And it's more efficient than a traditional motherboard. That process is sold out is highly constrained.
20:44 We don't use it. Third limitation in the space is 3 nanometer factory space at TSMC. So this is the most aggressive factory so with the most advanced technology. Our chips are at 5 nanometer so we don't use the 3 nanometer technology. And so >> And and then that's so fascinating so to ask the layman's question, so this this this What What does that mean stream military factor? This is a factor that's solely dedicated to >> That's right. They They build factories in the factory.
21:18 Etches transistors that are a certain distance apart. And the smaller the distance apart, the more transistors per square millimeter, and the more you can do per square millimeter. And so, the history of the computer industry making chips has been we've gotten better and better at putting transistors closer and closer. >> Yeah. >> And so, we used to be at 16 nanometers apart, and then we went to 10, and to 7, and to 5, and to 3, and they're working on 2, 1.8. And so, right now, the bulk of the the the GPU world is on three, and there's tremendous congestion at the factory.
22:07 >> And that's literally different tooling at the factory. Tooling at the factory. >> And we use the 5-nanometer factory, and this is our chip here. >> Yes. >> Why is it camera? This is the largest chip built in the history of the computer industry. It's 58 times larger than a GPU. It has you know, two and a half or 3,000 times more memory bandwidth. and for AI, bigger chips process information more quickly, and therefore you get answers in less time.
22:45 >> And obviously, you don't want a bigger chip for a laptop or a cell phone or for lots of other things, but for AI work, big chips are undoubtedly the best way to go. >> Okay. That's All right, so let's so I'm I want to do a bit of a deep dive on that, I'm going to self, you know, in a second. but to close on this, so three bottlenecks, you also mentioned CPUs a couple of times in this conversation and that seems to be a theme around the emergence of like a CPU shortage as well. Is that so? One, is that true?
23:21 Two, what causes it? >> So, a gentic AI is a world in which AI doesn't just provide answers, it initiates action. So, that action might be go to a website, it might be learn gather some data from a website, bring it back, take another action. Those actions are done by CPUs. And so, as AI gets better and better at doing things, at making instructions, calls for things to get done, we're using more and more CPUs. We And that is driving up the consumption of CPUs and therefore the demand for CPUs.
24:09 And so, this sort of huge push for more CPUs is being driven by AI on machines like ours and GPUs doing agentic work and asking the the the CPUs to take an action, to go to a website, to order burrito, to find a piece of information, to pull it from storage, to all that work is being done by the CPUs.
24:39 >> Including in a system like yours. So, who do you think the system like ours is interesting? So, so, you what what does it mean? So, you got you have your your chip and CPUs on the side and and those will be assistant? >> It means that AI, when we do AI agentic, the the AI processor is like the brains. And the CPUs are like the body. They're taking action. They're doing things in the digital world.
25:08 Under the direction of the AI, which is running on the accelerator, on the the streamer system. And so, as we do more and more AI work, and more and more agentic work, we're making more and more calls to CPUs. And therefore, the demand for CPUs is through the roof. >> And CPUs experience the same memory shortages. Memory shortages they see the exact same number. >> That's right. >> Okay. Yeah, let's let's go back to the product today in a in a second, but let's talk about you and your journey a little bit for for people to have context. So, you know, obviously you you pulled an all of an incredible idea with perfect timing.
25:51 And I know I know it was a little little bit >> The The way you have perfect timing The way you have perfect timing is to have horrible timing for 10 years. That's how you have perfect timing. >> From a very interesting this point, so like you guys started building this, you know, bigger chip, focus on inference, you know, you know, over a decade ago, 2016. 2016, which you know, makes perfect sense in in retrospect, given how long it takes to build a technology like this. But like what what what was the the vision like in 2016?
26:25 >> Cuz it was like 4 years after ImageNet, deep learning was a thing. >> It was all it was all vague, and it was all nascent. >> Yeah, revolutions. And and that's why we we don't believe the right thing to do is to embed the the latest and coolest model into your circuitry. That's a mistake. You know, we're the fastest in the world of transformers. And our architecture was set before transformers existed. And it, right, we're the fastest in the world at diffusion before an architecture was set for diffusion. What you want to do is take the underpinnings of those so that when the market moves, you can be good at that as well. And >> Otherwise, because of today's short life, to the ex-question or or earlier, so that's what others do. They build the architecture of the model into the >> Somehow. But historically, that's been a a structural mistake.
27:17 >> And then a structural mistake. So, you're doing the broader or the total platform on the straight any model work. >> You you want to think about what is the underlying calculation at the underlying calculation at all this work is sparse linear algebra. Yeah. And if you can accelerate that, whatever the bottle makers invent, you can make faster. Okay? And that was our approach. >> Yeah. So, you started in 2016. You had a prior company that you sold to AMD.
27:48 What were some of the lessons you learned there that you took into Cerebras? >> I think the lessons are large and many. I think What What are the things with, you know, I guess experiences is another name for having made mistakes and learning from them, right? I I think You know, what we have as a team built dozens of chips over the past 25 years. And the returns to experience in chip making is an artist. And we built a a different type of computer at at SeaMicro.
28:20 A type of computer optimized for low power and optimized for a workload that was very different than AI for something like web browsing. And but the the fundamental underpinnings, the questions you ask as a computer architect are always the same. What What can I do to to to make this work faster? Right? And is there enough of it to make it worthwhile? These are the These are the two questions we asked.
28:52 should we build a part for it? Should we What could we do to build a a chip optimized for AI? And will there be enough AI so that you can build business around it? Those are the questions we asked in 2016. And you know, they the flip side of that was I don't know. It wouldn't it be a surprise if the GPU, which had been optimized for graphics for 20 years and been pushing pixels to a monitor, was suddenly good at a new world. It'll What What if that be serendipitous?
29:30 And we came to believe that that it wasn't the right architecture for it. It was just better than the CPU. They And that we could build an architecture that would be vastly faster. It would use less power. And could drive down the cost of of the app. >> And that what was the the journey? So So what was very early, but >> Then we spent time in the desert. >> What can you do in the desert?
29:55 >> Then we wandered Then we wandered in the desert. >> Yeah, but you know, maybe maybe walk us a little bit through those years for any area you know, founders especially when deep tech founders are listening to this. So What was the So first of all, what was the issue? Was it market timing? Was it the technology was not working? and then how did you go about it as as a team you know, getting your your your board and your investors and raising more rounds so as as you presumably didn't have the proof points that you were wanting.
30:26 >> So for us, we at the beginning we were honest with our VCs and told them we're going to attack a really hard problem. We We weren't going to build something that was a little bit better than a GPU. And that our idea our strategy was that you will never beat a great company like Nvidia by doing something a little bit better. See, that they're going to buy everything for less. They're going to have pricing pressure.
30:53 They're going to be able to bundle. And the right strategy would be to do something incredibly hard in engineering that was way better. 10, 15, 20, 30, 50 times better. But to do that, right? Ordinary and obvious paths are all closed. Everybody else has taken the bait. And so, what we observed was that speed in inference was going to be a function of memory. And that there were two types of memory.
31:25 There's this DRAM and HBM. And they can store a lot. But they're slow. There's another type of memory called SRAM. It is unbelievably fast. But per unit area, it can't store very much. And that for graphics, everybody had always used DRAM. They used HBM. And that the AI workload was fundamentally different.
31:56 In graphics, you move data to the GPU. And then you work on it for a long time. And then you send the result. So, the time spent in total of movement plus work is dominated by work. In inference in AI, it's the exact opposite. You move a huge amount of data, all the weights, from memory to compute. And you do one calculation to generate the next word.
32:28 And then you have to do it again. Yep. So, all the time is dominated by the movement of data. So, that's why GPUs have so much trouble being fast. Okay. So, we observed that if we chose a strategy using SRAM, we could be fast. But, then we have to overcome the trade-off with SRAM. It can't store very much. That led us to the solution if we could build a part vastly larger than any part in history.
33:00 Right, the size of a dinner plate. We could stuff it to the gills with SRAM. Okay. And thereby get the benefit of SRAM that it was fast and overcome the weakness that it can't store very much. And that led us to a strategy called wafer scale. This chip is made from a single wafer that it comes out of TSMC. >> Do you want to define what a wafer is? >> A wafer all chips are cut from a wafer.
33:24 A wafer is a circular piece of silicon that's 300 mm across. And the process of chip making stamps out like your mother does with a cookie cutter. stamps out chips. The biggest chip that had ever been built before us was 800 sq mm. 840 to be exact. And this is 46,000. So, we had to invent all sorts of new technology to build a chip this big.
33:56 Okay. And once you build it, you have to invent ways to power it, to cool it. There are no vendors waiting for you. >> Yeah, right. >> Because you look like nothing else ever made. And that took years. And it was a deep tech problem. It had never been solved before and we had a an 18-month period where we were spending 8 million a month and we couldn't build them. So, if you're deep tech founders, you got to have that 8 million a month.
34:29 >> Now, why is so much what was the cost just to understand how this business is? >> Because what everybody thought was hard, we solved quickly. And And what nobody else knew about, because they'd never actually done it, turned out to be really hard. You know, imagine I tell people imagine the first group that was going to climb Everest. And they're at base camp, and they're having tea with a group that just failed. And the group that just failed says halfway up there's this part, it's unbelievably hard. We couldn't do it.
35:03 Okay. Your team climbs up, makes it all the way to the top, comes back. They're having tea again. And the team that made it leans over the team that hadn't made it, said that part in the middle, that wasn't the hard part. Because nobody gotten past it. Nobody gotten past certain things, so they didn't even know what to be afraid of. >> >> And it was something a step of packaging. And that's how you affix a wafer to a motherboard, how you deliver power to it, and how you cool it.
35:37 And nobody done it before. And over that 18-month period, we became the best in the world at it. From approximately zero. And we did that by sailing again and again, and using good engineering methodology, and doing a failure analysis every single failure. So we failed differently. It's And then it again and again and again. And we told our board, you know, you We met with our board every 6 weeks, and yeah, this is the strategy. Nobody's ever done this before.
36:07 This is what we're going to do. And we had to invent new materials, we had to invent new techniques, we ended up building things that that everybody else said partners who could do. But in August of 2019, we announced we'd solved it. And it'll my co-founders and I, the first time it worked it was sitting in a tiny little lab and we couldn't believe it grew the first few things this tree we took like one war and we just stared at you know watching the server run is about as exciting as watching paint dry and here we were the five of us just staring at this machine not believing it might work.
36:47 >> >> And then we made it work. >> So was it was it a bigger moment that actually ringing the bell or that just quite a few events? Like emotionally. >> Emotionally it it it was a completely different thing. it was that we had made our idea work. And I think for deep tech founders it particular common there's always this little thing in the back of your mind that says and maybe it's >> >> Right, maybe we're actually crazy.
37:13 >> That's right. Maybe it's going to fail and maybe it's not going to work. And maybe I don't have time, maybe we're going to run out money, maybe maybe maybe. And the flip side of that is the joy that that this was my co-founders ideas. I mean these are their ideas manifest in the world. And that is an extraordinary thing. And it is when someone's ideas take physical shape. And then the next step is when you watch other people's work sit on the top of your idea. Then you know you love making infrastructure.
37:51 And so what when we rang the bell and we went public on May 14th of this year in the largest semiconductor IPO in history. what and we did something unusual we invited all our engineers who'd been with us since the start everybody who'd been with us more than nine years and their families and we all rang the bell together. We all got up on stage. that was sort of a moment of with pride that that that we had done this together. Like and that we should manage not to die.
38:29 >> >> And sometimes when you start up, you know, people aren't honest. Mhm. They don't tell you that, but we we'd avoided some We made plenty of mistakes, but we've been avoided the fatal ones and we we made it through to a a level of success that gave us the opportunity to to pursue a new level of success. I mean, that's what an IPO is. It's not the end. It's you've gotten to a plateau that you can chase a new level of success, see, in the public market. And and that felt pretty good, I'll admit.
39:04 >> Amazing. Amazing thanks for sharing. So, just heading going a little deeper on some of the product and and technical stuff. so, one of the trade-offs of building a bigger chip, I mean, one that may come to mind is failure mode. >> Sure. >> If you have all of little chips on a big wafer, for example, you can isolate the problems. Yeah, if you have a big one, then everything will fail at the same time.
39:38 >> So, we actually thought carefully about failure mode and and we we invented a technique that had a kind of about a million identical tiles. And if one fails, we can shut it down and we can use a redundant one. And we should go. Hey. And so, if you're going to go big, you have to think about in the very architecture of the computer, how you're going to manage failure. I mean, the GPUs have a huge failure rates. So, I'm sure you guys have spoken about this. Infant mortality is enormous and they fail all the time. There's good data and Facebook put out a paper on the number of failures that you get in a big cluster.
40:22 >> Now we can do some other things because we have all this compute in one spot. >> We can invest more to cool it. >> So we pioneered water cooling in AI systems. >> And we want these much colder. >> And GP ends. >> And the failure mode in electronics is temperature. >> And so by running them colder, we are more reliable. >> And so that was some an advantage. >> But again, we had to invent the technique to to to cool them even at this big.
40:56 >> Yes. >> And so I I think we we have a a system mentality. Right? If you're going to do something big, you've made trade-offs. And you have to think about an architecture that allows for redundancy and repair. You have to think about the pros and cons every aspect of the architecture. >> And and that's how you do something different and new. But >> And and again just to drive the the point of home to make sure this I guess clear takeaway you know for people from from this conversation. So explain it to me like I'm maybe not five but 15.
41:37 Why is this faster than a GPU? Like what fundamentally makes it faster and what country is super fast with the GPU? >> How tokens are generated in inference is why it's faster. In a In an inference to generate a word and our answers are a whole stream of words. They can be code, they can be pixels, but call them words. >> Talk. >> The weights of the model are moved from memory to compute.
42:12 A calculation occurs. And that generates the word. >> Is that prefill versus decode? >> That is both steps. >> Okay. And you want to maybe explain what prefill and decode are? >> Okay. There are two steps in the computation necessary to do inference. When you type into chat GPT, it's letting me the history of this village prior to World War II. And it can't see you. Yes.
42:46 Two things have happened. The first thing is your prompt has been processed. That's step one. And step two, your answer has been generated. >> So >> And the way your answer is generated is called decode. >> You got it. >> And decode is sequential. Processing your prompt, which we'd call prefill, can be parallelized. So you can process many of them simultaneously.
43:23 But the speed with which you get an answer is a function of the decode. And it is step by step by step in sequence, and that can't be changed. And how you do that step is you move weights, which are the intelligence from the model, to compute to generate a word. You do a calculation, and you get a word. And then that word is used to generate the next word, where weights are moved from compute.
43:56 So the process is one of moving weights from memory to compute. So how big are the weights? In a little model, like a 70 billion parameter model. The weights are about the size of 100 HD movies. So, to generate a single word, you move 100 HD movies from memory to compute. And then you have to move them again for the next word. And you want to do this a thousand times to get a good answer. A thousand word answers.
44:28 This is where HBM is slow. This exact step is where HBM is slow. And that exact step is where by having all the SRAM share, we're blisteringly fast. And so, the speed of moving weights to compute is about two and a half thousand times faster here than on a woven GPU. And so, that's the essence of what we've been able to do here and why it's so much faster.
45:01 >> Fascinating. Do you think that models will need or will evolve as well in that fast AI world? Or is that just a cheap problem? >> No, I don't believe so. >> Because remember, two things are happening. One, we want the models to be smarter. And two, one of the ways models are getting smarter is with RL. And RL uses inference inside of training. And so, the faster you can do the inference inside of training, the faster you're >> training.
45:35 >> What are the things that is that a market for you guys in the >> Yeah, that's a market for us as well. >> So, you're not just >> Inference is so as we do RL and we do traditional training, too. Not for the largest models, for the largest lab, but for the next tier. >> Or pre-training for pure pre-training? >> Pre-training. >> fine-tuning. >> A full set. >> So, in short, you pre-training and post-training was RL some pre-training for the other labs.
46:07 what I mean do you use ST all better than you for what job that >> So, GPUs is in training have some challenges that have been solved by a very narrow selections of the community. GPU is a very small chip. And the calculations that we need to do in training are very large.
46:39 And one of the most complicated parts of training is the breaking up the calculations and spreading them apart on multiple GPUs. And that's called distributed compute. That has historically been the domain of the supercomputer world. And is very difficult. Not just because cracking a problem and having lots of others work on it is hard, but they have to constantly share information. And that sharing information is why they needed to buy Mellanox. They is they needed to control a fabric over which all the sharing in order to get an answer would happen.
47:20 That breaking up the a big matrix multiply big calculation is called running tensor model parallel. You're breaking up the tensor and spreading it apart. And and the best labs in the world are good at that. But nobody else is. They They When we run training, we don't have to run that way. We run what's called model parallel we data parallel. And data parallel is very simple. And so it allows teams who are good and very good to quickly test ideas in training.
47:57 And so we are easier to use and faster. because we allow them to use a technique which is much simpler. >> Let's talk about the intelligent world. Maybe starting with with with reasoning. I think I saw a blog post where you guys have ever given a statement that reasoning was was not always the right solution for all problems. Like how do you how do you think about this and where does that fit in the answer to the world?
48:30 >> We're using this as a technique. a little bit like when you were in eighth grade and you wrote different drafts of the paper. single shot inference is a seat. You get an you know you you write a query, you get an answer. The easiest way to think about reasoning is you're going to do several drafts. It's going to break the problem up.
49:00 It's going to solve it in parts. It's going to bring them together. It's going to review the results. It's going to improve the results. And then it's going to give you an answer. And so that is going to take more compute. And if your computer is slow, that's going to be a more than an irritant. That could be crippling speed. And so as the best models, all of them, whether they're domestic or Chinese, whether you're OpenAI or Anthropic or Gemini, whether you're any of the Chinese models, they all moved to a reasoning approach.
49:38 But that meant more compute was being used during inference. That was a huge advantage for us because we were fast change. And it made the GPU slowness stack up. And it made our speed have an even bigger advantage. And so this was a huge building for us. We think this is a going to stay as a fundamental essence of the way these models are are are run right now. Right?
50:09 >> And you also wrote about verification and whether what was the bulk bottlenecking agents was whether the reasoning was good enough, whether they were smart enough, or whether there was a verification problem. >> Sure. I I think the the verification problem is a little bit like the guardrail problems. and what you'd like to do is after you've written two or three drafts of your answer, you would like to be sure that it wasn't wrong.
50:47 And that's your verification step. And you can do that maybe with a different model. You can do that by asking your existing model a similar question in a different way. Right? All of these are ways you you can pressure test your result. guardrails can work the same way. Right? You want to review either with a model or with another technique through a scoring mechanism that this question isn't out of line, that this answer isn't about how to make biological weapons or calling upon information that that that you are directing the FBI.
51:33 Right? and all of that takes compute time. And so whether you're trying to improve through reasoning or whether you're running guard rails by being faster. All right, you can get results in less time. >> Hey. So what do you think the world is evolving towards? Is it a bunch of smaller models running faster doing more verification versus a large model?
52:03 >> Well, I think those work together. I think your big model produces an answer. Then you want to double you just want to check your data. I mean in in the journalism industry, people would write papers that have data checks. Right? Somebody would go and make sure back in the world journalists check data. You had a data checker, right? That was a job. Each claim was checked independently. that's a different model. Right? The big model wrote the piece and then a little model checked some of the answers. And I think that's a very a very good way to go by.
52:37 >> Where does our multi-modality with the larger models falling in your world? >> We just announced that sort of that we were fastest in the world on on one of Google's multimodal models. I I think the truth is is that there's very little text that doesn't have charts and graphs, right? You You You must to understand text be able to understand illustrations, graphs, charts.
53:11 And so that that's sort of that the first and easiest part. And then you want to be able to create >> Hey. >> And then you want to understand images. And I I think the the new models are very very good at that. Obviously what follows that is video because a video is just a collection of images. but that takes an enormous amount of compute right now. And that's one of the reasons it's been sort of set aside by deleting labs. They get so unbelievably computation intensely.
53:50 >> Great. Let's talk about this business a little bit because you guys obviously chip maker and provider as we discussed. You're also a cloud provider that doesn't a provider and what's So, what are the different parts? >> We make compute. And that computer is optimized for AI. It is the fastest in AI world. If you have a data center, we will sell you hardware for deployment in your data center. If you don't have a data center and you'd like to rent it by the month or the year, we have data centers.
54:28 They have so you can rent our our equipment through our data centers and through our cloud. And so that allows us to get AI natives as well as large enterprises and and governments. >> And what's the percentage of share in the And also the >> It was 50/50 last year. and I I think this last quarter it was maybe maybe 75/25 in favor of of hardware sales. I I think this year it might be 50/50 and as our Open AI deal continues to unfold, it will probably be 30/70 with 30 on-premise deployments of hardware at a 70 cloud.
55:14 >> Great. And let's talk about that Open AI deal since it's such like a major historical milestone, record making. so, it's it's providing up to 750 megawatts. Which is Which is interesting by the way as a as a metric because I mean a chip provider, but this is power. So, is that short a hand for us? >> It's a short hand. It turns out right now, and we didn't talk about this because there's sort of in the adjacent supply chain. We went through the shortage of of memories. We went through the shortage of a process called Co-Op 3 nanometer capacity.
55:50 The The other limitation in our industry right now is data center availability. And that is a limiting factor for everybody. And that's why Anthropic did a huge deal and sort of very expensive deal with the odds. I had for data center capacity. our deal with them with with Open AI was because data center capacity is a limiting constraint measured the way data centers are measured in in megawatts. They The deal is 750 megawatts, 250 megawatts in '26 on a multi-year lease.
56:34 An additional 250 megawatts in '27 on a multi-year lease and an additional in '28 and multi-year lease. >> And they're doing the data center for them or providing the chips that are going to the data center. >> Data center for them. We're delivering a full cloud solution. So, they connect to us via an API. >> You said you mentioned '26, '26. Immediately that's telemonics. I see. >> I I am looking for data centers. My next meeting is in the with a data center provider.
57:05 >> We're doing a lot in Europe first now. >> Oh, it's just a lot in the in the Nordics. >> Is that's because it's closer to power sources? >> Yes. It's because there is low cost power. How about that? Clean low cost power and low cost cooling. >> Hey, doesn't matter just talking in the time business where the data center is is is located in terms of >> >> Spain. >> Yeah, there is an additional latency called transport latency, and that's the speed of light through fiber hurt to get from Helsinki to New York. And you have to account for that if your customer is in New York and your data center is in Helsinki.
57:50 Yeah, and it's usually about 2/3 the speed of light. In case how long it takes you would like data centers on the same continent. Both of mine >> Hey. That's the opening ideal. there was an exciting deal with AWS as well, where you it's it's a it's a co-chip solutions, right? >> It's the disaggregated solution you mentioned before, where their training part is doing the pre-fill. And it is doing the parallelizable step.
58:23 So, as training as training is doing the the pre-fill step, then our chip is doing the decode. And so, you get a lot more bull shittably fast tokens. And it's a good deal for us. It uses their data centers. So, they use their deployments in the AWS data center. >> Yeah. It's fascinating about this industry is how it seems that flexibility is so important. There's just something about this solution like you need chips, you get chips, you need data centers. Everybody's buying from different suppliers to reduce dependency.
58:57 >> I I think that's one of the reasons why we we need went from being a traditional chip consistent provider to also offering data centers is that what what our customers want are fast tokens. And anything we can do to make these delivery of fast tokens easier. For some of them, that's in their data center. For some of them, it's with an API. Just point your traffic to us. And we'll point the fire hose of a fast token in the back.
59:28 >> Yeah. And this token is a sense for where where where you start and where you stop. Is there do not or Both. at least currently provide the the the coverage. So if I want to run Jimmy, like I know you have like incredible stats for QA and Gemma and those being so forth and so forth and so forth. but you don't run those as a service, do you? We do. We do. Okay. We So you have a service where you're competing with the base dance and fireworks. Yeah.
60:00 I I I think on Cool. We have a an on-demand service. where you can come to our site and and book a month. I think you can even buy buckets of tokens for for Kimmy or GLM or some of these models. many of our customers come there, get excited about it, and then move to a dedicated offering. Where's they take hundreds of machines for a year or two or three or four. It's once they've proven out the benefit for them in their work.
60:38 Often they do AB tests. Not surprising, people like faster. >> You got So so what is more like a testing >> It's a full environment. >> Okay. >> You can you can go and use it. It's at servers.ai. Go play around. >> And fascinating but that could become like a yet another big cloud business. >> We're the master. >> Okay. So you have chips, you have data centers, and you have a cloud business running inference on governance. It's going to be very fascinating.
61:05 Thinking about modes, so you know, famously Nvidia has CUDA as well discussed mode. What's your equivalent of you know? >> I don't think CUDA is a mode. >> Right. >> Really, we should talk about that. >> Yep. >> I think 2 years ago every state-of-the-art model was trained in a CUDA flow. And right now Gemini is trained without CUDA.
61:36 Anthropic Claude is trained without CUDA. OpenAI is trained with CUDA. So, in a one or two-year period, they lost 70% share of training models >> Or is it the state of the art? >> all because Gemini is trained on TPUs. It's very bullish. I don't know. Anthropic is trained on training. Not H100. Eight. And so, the I think the the storage of the boat it is still present where the data showed the boat is clearly shrinking.
62:16 there's no boat in inference. it takes you eight keystrokes to move from a GPU to us in the cloud. So, it is four keystrokes. >> Oh, eight keystrokes. So, it's eight keystrokes. >> Eight keystrokes to work. That's it. To move your traffic from an GPU API to us. And so, obviously CUDA was sort of enormously important in the creation of our industry. And in allowing the the graphics processing unit to be more general than graphics processing.
62:51 But, since '23, '24, quick wide, I think it's ability to solve the durable mode has shrunk substantially. >> And you have a whole ecosystem strategy. As you think about your mode, to the extent that any mode can be frozen in this industry, right? You can build a whole ecosystem. Is that Is that >> Yeah. Yeah, we built an ecosystem. I I think our mode comes from fact that by virtue of our architecture, we are doing things no one else can do. It's It's And it it's not that they can spend more money or or or they can't pay more for for these who keep you. If you want fast, you can't have Well, we can't give it it doesn't work.
63:43 And so, that's where we're building the sort of our strength, and that's how we're we're sort of delivering value to customers. >> Anything about supply chain We mentioned supply chain constraints for others, but what are your supply chain constraints? Are you Are you old TSMC? >> Near TSMC. we have very close collaboration. They were investors of us. They've been exceptional partners. Come on, man. I'll tell you an unusual story in 2017.
64:17 We showed up, and we were about 30 guys total. And we showed up in August. >> Horrible time in Taiwan. Don't go to Taiwan in August. It is brutal. >> Not that it's so nice here today. >> >> It's only 90 here. >> You're trying to embarrass me. >> Yeah, but all this humidity again. Yeah, you got to put on a suit. And we met with the leadership of TSMC. We said we we would like your little pipsqueak company We We believe we can solve a problem that nobody solved in history.
64:52 And here's how we would modify the way you make chips to make this possible. They thought about it. And they said, "We agree. Let's do it." >> In the meeting. >> In the meeting. >> In the meeting. It wasn't go away and for a month in the meeting. >> Was that a matter prepared mind or they were just exceptionally fast on their feet? >> first the salesperson had gathered the decision makers.
65:22 Second, we >> Go. >> Our our proposal was sort of really good at allowing them to use what they were good at. And it didn't require them to change a huge amount. But it it did require them to make real changes. And I think they saw this as sufficiently bold that they would learn as they did it.
65:56 And they also knew that AI was better on the base hooks. Okay. And so the combination of fair mind, a willingness to take some risk, bold thinking from a a very large company. >> Cuz that makes It is fascinating. I mean, that's how big companies win. >> That's right. >> and how rare is that? It was extraordinary. >> And and what happened next? Like how long does it take between the decision in a meeting like that which sounds as exceptionally fast to >> 2 years.
66:25 >> 2 years. >> But chip making is a long hard process and and most of the time your first chip isn't a winner. And there are lots of of startups now and some of them with really smart guys. >> Yeah. >> But their first chip will not be a winner. >> Mhm. >> the TPU Google had some of the best guys in the industry. First chip wasn't a winner nor the second nor the third. Fourth was really good. Now they're on their eighth and it's really a great chip.
66:53 >> Okay. >> the analog team at at AWS first chip wasn't great. Second, third chip were really good. It It takes time. And so we built a chip, we delivered it. And this gets to an earlier question you asked, you know, we solved a problem that that nobody in the history of computing solved, and we delivered it in 2020. And nobody cared. Nobody cared. Nobody bought any and nobody cared. >> >> Which you were like, "God."
67:26 >> Like, nobody Everybody said we were crazy, it would never work, and now it works, and like nobody wants to buy it. >> It is Yeah, it is. How old is >> So, then we built the next one. I mean, the first one we probably sold 20 or 30. >> And and nobody wanted it because the market was not ready or because the product was not good enough? >> Nobody wanted it because AI was a hobby.
67:50 It was Right. And who cares if your hobby is really fast? So. you care about fast when it's in production. You care about fast when you use it every day. And so, we built another one, and you know, that one we sold three or 500. >> Yeah. >> Yeah, we built the third one, and we sold tens of thousands. >> That's amazing. >> Yeah, isn't that interesting? >> And so, going back to supply chain, do you need to think about onshoring, diversification?
68:24 >> So, it's very hard to diversify away from TSMC. Chips are so hard. And you actually, when you design a chip, part of the design is for the rules of the factory. Right? So, you can't take your design from TSMC and go to somebody else because a huge amount of the work was done to be sure your design is within their rules. And so, England, I think, only with one or two exceptions in history, each chip generation goes to one fab.
68:58 So, that that's we're going to be with TSMC for our next generation as well. >> I think some of them we have a supply chain that is built in many parts, but we bring the chips back from TSMC to the US. We package in the US and we assemble in the US. We do our manufacturing in the US, so then we ship from the US. I think when you're growing this fast as we are right, there are a whole range of garden variety supply chain challenges.
69:33 You know, a vendor screws up a batch. It gets stuck in customs. The number of ways that that things can go wrong in the supply chain is unbelievable. But we manage these every day and we're increasing our manufacturing throughput exponentially, and so it's really that part of the business is going very well. >> Incredible. So maybe to zoom out as a last question what what's your best guess about how where all of this is going? You know, obviously who knows in the eye in the next like few years, but like in the next year or two >> Well, we know some things. We we know that the model you use it you're using today >> stable or or GPT-56 >> will be the worst model you ever use.
70:20 And whatever you think is cool about it right now is going to be boring and backwards in 6 plus. And that is so exciting. And I I think I watch the way our young engineers use it. And it's a very different than the way I'm using it. And it's just such a fun time where you can learn from your young team members. They're using AI very differently. I think the business of dashboarding and the business because AI doesn't the damage it's doing to SaaS is that I think unrepairable.
70:57 I long he's showing you could ask your AI build me tool like Salesforce. 30 seconds later you have a working tools that they is just unbelievable. And all the things that that were difficult because they cut across your inside organizational silos, right? One of the things is really hard if you want to know for your top performing people when was the last time they got a stock option refresh and how much holding power is left. So how much uninvested stock they have left at today's price.
71:34 It was like five systems. You're in Workday. You're in your stock. You're in your your comp. You're in Carta. None of them can have and that's what the CEO wants. How much holding power is for my top guys? And I used to have little tools I wrote for this and I've got a a little little app that that I had and Spark writes one. You go. >> >> Well, what what a story. It's just incredible to share all of this from you. What a journey and what an exciting future. So thank you very much. I learned a lot and this was terrific. Thank you Andrew. Thank you for for having me on your show. I really appreciate it.
72:22 Hi, it's Matt Turk again. Thanks for listening to this episode of the Mad Podcast. If you enjoyed it, we'd be very grateful if you would consider subscribing if you haven't already or leaving a positive review or comment on whichever platform you're watching this or listening to this episode from. This really helps us build a podcast and get great guests. Thanks and see you at the next episode.
Summary
- Cerebras has developed the largest chip in computing history, which is 58 times larger than a GPU, optimizing it for AI workloads.
- Speed is crucial for AI inference, with a focus on delivering fast tokens per second for productivity.
- The chip industry is evolving from general-purpose CPUs to specialized chips (ASICs) for specific workloads, with companies like Cerebras leading in AI optimization.
- Current bottlenecks in the chip supply chain include memory shortages and manufacturing constraints at TSMC.
- Cerebras' architecture utilizes SRAM for faster data processing, overcoming limitations faced by GPUs that rely on slower DRAM.
- The company has transitioned from a traditional chip provider to also offering cloud services, allowing customers to rent hardware for AI applications.
- Feldman believes the AI landscape will continue to evolve rapidly, with future models surpassing current capabilities and transforming industries.
Questions Answered
What is significant about the largest chip built in the history of computing?
The largest chip, built by Cerebras, is 58 times larger than a GPU and is designed for faster AI processing. Despite its groundbreaking capabilities, it initially faced skepticism and low market interest.
How does cloud computing relate to AI processing demands?
Most real compute tasks for AI will be handled in data centers rather than on local devices. Cerebras focuses on data center compute to meet the high demand for AI processing.
What challenges did Cerebras face in developing their chip?
Cerebras recognized that simply improving GPU technology wouldn't suffice against established competitors like Nvidia. They aimed to create a fundamentally different architecture, which led to a challenging development period.
Why is speed crucial in AI model processing?
The speed of moving weights from memory to compute is critical in generating AI responses. Cerebras's architecture allows for significantly faster processing compared to traditional GPUs, enhancing overall efficiency.
How does flexibility play a role in AI chip and service offerings?
Cerebras has adapted to provide both chip solutions and data center services to meet diverse customer needs, focusing on delivering fast AI processing tokens efficiently.