The Silicon Gold Rush: How AI is Driving the Development of New Chips

1:39 on the on the Heat. Heat. N.

2:49 >> >> Hey, apple. >> >> for me.

3:48 Hey Come on. Come on. Come on. Hey.

4:22 Hey. Hey. Heat. Heat. N.

5:01 >> >> Hello. Hello. How's everybody doing tonight? >> Yeah. Yeah. Great to see everybody in person. Great to see everybody online. I'm Mark Edkind. I'm the CEO of the Computer History Museum. One of our goals at the museum is to have legends and leaders on this stage talking about the most important topics in computing.

5:37 And that's what we're doing tonight. We have three leaders who are leading in the future of chip design for artificial intelligence. You could say we're in a Cambrian explosion of chip design right now. We're going to hear from the leaders of that revolution tonight. I want to thank our sponsors. it's Mark and Mary Stevens. they are sponsoring these great discussions, not just tonight, but they sponsored a whole series of discussions. and really thankful for them for their leadership.

6:07 We want every one of you to be part of the conversation. So, you're going to see a QR code on here. If everybody holds up their camera, if you get that QR code, you'll be able to respond to polls. You'll be able to ask questions as well. so please grab that now. And then lastly I want to introduce the leader of tonight's event Dave Patterson. Dave is the party professor of computer scientist ameritus computer science ameritus at University of California Berkeley. he's also a Google distinguished engineer. you probably know Dave as the leader of the risk project. We know Dave here at the museum as a huge friend of the museum. he has been a fellow which is our most distinguished honor. and he has shared his collection with us. and so we have some original pieces that are important for our collection to help educate the next generation. And most importantly, he has been a friend to our curators. and he has been advising us on our collection and our storytelling.

7:11 and he's a frequent guest on our stage. So please welcome Dave Patterson. Thank you. >> Thanks everybody for showing up. I get to introduce two of my friends. Bill Norm and I have known each other since the 1980s when I was an assistant professor at UC Berkeley and Bill was a grad student, PhD student at Caltech and Norm at Stanford University. Bill Deli has been building network connected computers ever since Caltech. after graduation he joined MIT where he built a couple of more network supercomputers or network computers. after about a decade he came back to Stanford University.

8:00 where he eventually become chair of the computer science department at Stanford. In about a decade later in 2009 he became the chief scientist at NVIDIA. That was 16 years ago. He still holds that position. But now he is also a senior vice president. I was trying to figure out how to figure out of all of his contributions what to pick. So right thing to do is ask AI, right? So the AI chatbot according to the AI chatbots his most significant contributions are a few but it's wormhole routing and related flow control techniques and it's a practical low latency switching method for chip and system level networks and he's also wrote a book with his friend Brian Tols principles and practices for interconnection networks. This is kind of the bible of networking which systemized topology routing and flow control design and it shaped both academic research and industrial research for decades. Also at Stanford his imagin and Marrammac projects kind of made a very strong case for stream processing which was an important conceptual precursor to a GPU modern GPU computing.

9:15 Norm when he was a graduate student at Stanford worked on the MIPS project which is a pioneering risk architecture. After graduation he joined digital equipment western research labs where he stayed for about a decade and then he went to HP labs. After that in 2013 his friend from WRL Jeff approached him to try and go to Google to build hardware for this new deep learning machine learning ideas. Norm was very skeptical given all the hyper all the hype about AI in the past but Jeff over kind of convinced him to come because he said this deep learning it works on everything we try. So he joined 2013 he is now both a fellow and a vice president at Google. I asked AI about Norm's contributions and they mentioned the TPUs for his work at Google and then his work on memory hierarchy design at deck and HP particularly victim caches and prefetch buffers. As I prepared this Norm and Bill have these incredible impact and highly parallel lives so by 10 features that they're have similar careers. So the first is they got their bachelor's, masters and PhDs at three different institutions. I was just a stay-at-home state at the same one. Secondly, they both got graduate degrees from Stanford.

10:42 masters for Bill and a PhD for Norm. Both of them were supervised by John Hennessy. So who I think is here tonight. both are vice presidents at their respective companies. Both are fellows of ACM. Both are fellows of E. Both are fellows of the American association for the advancement of sciences. both were elected to the national academy of engineering. A very prestigious honor. Both won the ITE Seymour Cray computer science and engineering award and both won the ACMI Eert Mockley award which is the highest award in computer architecture.

11:20 So they took they have very busy lives but they took some time out to be with us. So, I'd like to let's welcome them with a round of applause for their contributions and for being here. Great. Thanks, Dave. So, for about 30 or 40 minutes, I'll do the we'll do a question. I'll do the questions, but after that, we'll open up to audience questions. So the title is Silicon Gold Rush which suggests a period of intense innovation, investment and competition. From your perspectives at Nvidia and Google, what defines this gold rush era for AA hardware and what makes it fundamentally different from previous eras of computer architecture development? And start with Bill.

12:10 >> Yeah. So I think there are three characteristics that really define this one and make it very different from some in the past and it's really intense economic demand there. you know AI is working on everything they apply it to like Jeff Dean said and as a result the demand for you know more more tokens more flops you know more AI is insatiable at the same time the application is rapidly evolving but relatively simple compared to the big supercomputing applications we had with millions of lines of code a transformer is a relatively simple thing and so it's easy to look at what you have to build to make that go fast and and also people are willing willing to evolve very quickly. There are no dusty decks. If you want to look at contrast, in the early 90s, there was sort of a computer architecture gold rush around supercomputing. It turns out there was no intense economic demand. It was fueled by the the the DARPA strategic computing program. They gave a bunch of funding, you know, money, made people think that there was a big downstream market, which there wasn't. Lots of people founded companies, which most wound up going bust. things like thinking machines and and and the like.

13:16 the applications then were dusty decks with thousands of lines of in some case close to a million lines of code and you could accelerate you know what appeared to be the important kernel but you know Anvil's law would come up and bite you because it was the other 99% of of the code that wasn't accelerated. So the ease of actually getting things accelerated with a simple you know application that you can accelerate a few primitives and get really good results real economic demand I think make this one this one stick. This is not one of these, you know, like the, you know, list machine frenzy of the 80s or the supercomputing frenzy of the 90s.

13:52 This is a real value delivering event. >> So, so actual market demand for computers, Norm, what what's your take? Yeah, I I'm kind of a fan of a glo keeping attention of what's happening in the global economy and the amount of money that's being invested is just in incredible. So, you know, for a while New York had a a leaky tunnel that was 100 years old going to New Jersey and they they were a billion dollars short and then you know like Google has announced they're going to have $105 billion worth of capex next year and other companies are investing similar amounts. So, it's yeah, there's going to be winners and and losers and everyone wants to to race as fast as possible.

14:49 >> Yeah. Well, you were you guys were making an analogy to the actual gold rush. So who who's the who are the prospectors and who are the sellers of the pickaxes here? >> Well, the prospectors are the people who are, you know, finding verticals, right? And if you find the right vertical to apply AI to, I think you can strike it rich just like the prospectors did. you know, Nvidia and, you know, companies like Google that make the infrastructure. We're Leel and Stanford, right? We're selling the the picks and and shovels to the prospectors and whether they strike a rich or not, we're going to make money.

15:22 >> Okay. >> >> Well, well, we're still both paying the memory companies a lot of money. >> Yeah. Yeah. Yeah. I think it was Micron made as much money in the last quarter as they had made in the 19 years previously. So, >> it's interesting seeing their revenue go up while the number of parts sold stays level. >> Exactly. Yeah. The margins through the roof. Yeah. So an unnamed colleague of mine has accused training accelerators of convergent evolution. So that's a biological jargon for independent evolution of similar features and species of different lineages. So his convergent his claim that you guys have a convergent design is a max radical compute die with a large systolic matrix unit that's you pack as many HBMs that fit around the outside and the perimeter and then you use the fastest 30s you can to do a custom link. I I I expect you don't agree with that and so why is he wrong and that GPUs and TPUs have divergence approaches and is there anything you like about the other design?

16:34 >> Well, Norm first I >> I'll start. so one has to remember about the Cambrian explosion. There was also the implosion that happened afterwards. So there were all these weird and wonderful creatures, you know, that look like giant pill bugs but with like weird appendages sticking out. And they they didn't survive the extinction event. And so I I think this the same thing is true.

17:06 There's been many startups with many different ideas, but I I think the reason why Nvidia and Google have both been successful is that a bunch of these other ideas weren't as capable and so you know they may go extinct. >> So that that's why it converged. You both had good taste in what you chose to >> Yes. But yeah, I don't know that we've even converged.

17:41 >> Okay. >> I think I think TPUs and GPUs look look very different. I guess that >> Okay, great. >> at a 20,000 foot view. Yes. You know, the application drives certain requirements, right? You have gems, you're going to need some matrix multiply units. You have, you know, softmax and and norms, you're going to need some vector units and and things that can do transcendental functions. You need a certain amount of memory capacity, a certain amount of memory bandwidth. need a certain amount of communication bandwidth, but there's a lot of nuance, right? That's what the application demands. Everything is going to have that, even though weird bug-like creatures are going to go extinct.

18:11 but there's a lot of nuances in how they're how they're combined. For, you know, many years, Nvidia, I think, kind of led the way with improved numericics. you know, we we published a paper in MLYS a few years ago and talking about vector scaling and out of that came NVFP4 and then a few years later all the MX FP formats kind of kind of followed suit. and and that that's an important nuance. It can give you a 2x advantage over somebody who who was doing just FP8 before to get the same level of accuracy. Similarly with sparity sonhan and I wrote a paper in 2015 in nurup's about you know you know most neural networks are naturally very sparse and then we put hardware support for sparsity starting in our amper generation that's something you don't find in every one of these bug-like creatures there's a lot of a lot of nuance down the way now I think Google has a big advantage in in designing their TPUs and up until recently they had one customer Google and so they could decide exactly what they wanted to do and do it you know and Nvidia is fortunate that we have a 68% share of both the inference and and and training market. But what comes with that is we have lots of customers and we have to keep them all happy, which means they come and ask for feature feature requests and if you're a big enough customer, you have to at least kind of entertain it a little bit and make make it seem like you >> so you have you have to do a lot of travel and talk to them >> and and you you have to, you know, put things into your hardware to make all these different different people happy.

19:32 And I think that that drives certain certain things in the machine that if we if we could just build exactly what we wanted just for ourselves and not to listen to customers, we might do better. So I'm a little bit envious of the TPU's ability to u to do that. >> So So you admire their small customer base. >> Yeah. Yeah. I I I do think that you know one one thing that they have is they have they have a they have a pretty neat interconnection network with the with the training TPUs of the 3D Taurus. I built a lot of supercomputers, you know, both at MIT and with Cray in in the '9s with 3D Taurus networks and so I'm very fond of that and I can see the hand of the co-author of my book Brian Tools work in that network >> and we studied that book very carefully.

20:14 So in terms of the Cambrian explosion though I like liked your analogy there. You know there was a similar Cambrian explosion with graphics chips. >> I don't remember when exactly there was a sort of late 90s early 2000s there were no fewer than 100 graphics chip startups in Silicon Valley and you know after you know you know whatever the weeding out of the not fit you know the survival of the fittest there were two remaining. It was Nvidia and ATI which wound up getting getting acquired by AMD. And I think something very similar is is the likely outcome of the current explosion.

20:48 >> Okay. Norman is he he likes the and I agree. I think the optical interconnect >> that Google I of course >> I'm biased because I get a paycheck from Google but I think the optical interconnect is a very cool feature. What what about the Nvidia architectures you find attractive? or or the customer base or what >> I think Bill said it well in that you know there's certain operations that we both have to support you know matrix multiply vector operations one of the things that was different about TPUs is I mean our first one was a PCI card but and just did inference. But starting with the second one, they they were designed to be supercomputers. So we had the the tourist network as in in Bill's book with Brian and so we didn't have any painful steps along the way of adding features and having other features kind of conflict with those features and stuff. So we were able to start out with a clean sheet of paper with a a supercomputer design.

22:17 >> Yeah, I think that was a precient that the fact that your second design in 2017 you realized that you needed a supercomputer for training and that's what you were going to build right from the start. >> Yeah. >> And it was it it was and liquid cooled in 2018 or so. >> Yeah. Yeah. So, we've been liquid cooled for eight years. >> Yeah. So, and that's what a lot of these look like. Okay. and by the way, I'd like to thank the people at Hot Ships, which happened right here in Silicon Valley a couple days ago. I went around and asked for questions to to give Bill and so I got some really goodness. Okay.

22:53 Nvidia and Google are both worldclass system companies. Has the engineering of systems, the interconnect, the memory, power, cooling, and packaging become an even bigger moat for competitors than the silicon itself? And how do your approaches to interconnectivity address the critical bottlenecks and scaling? Where are they similar and different? I I think Bill, it's your turn to go first. >> Okay. So, I I think the product is the whole system. That's, you know, not just the GPU or even the board with GPUs and CPUs and networking gear. It's all the hardware, all the software, and the configuration that makes it all work well together. Starting, I think, in our Pascal generation, I think it was about 2015, we started offering something we called a DGX super pod, which was if you bought this, this is even before we had all the captive large scale networking.

23:41 We had our own internal, you know, MLink networking, but we we hadn't yet acquired Melanox. >> and but we, you know, picked, you'll use this switch, you'll configure it like this, and you do that. And if you did that, you could put together, you know, a collection of in that period of time maybe 10,000 GPUs and you would turn it on and it would work. And in contrast, you know, when we would build the big DOE supercomputers like when we did you know, Titan in you know, what was that, you know, 2011 or 2012 and and Summit and Sierra, you would typically go through a six-month period after you had all the hardware working bringing it up because of little things in network configuration, configuration of the different things.

24:20 So you're delivering the entire product. You're delivering a system. It has to work reliably over long training jobs with very high availability. And so you know I think there's a lot of systems expertise that goes into delivering that and having a standardized configuration for us being because we're selling into a lot of different data centers where people did things different ways was critical to get the system up and running quickly and and run it extremely reliably. Now you know where are things different? I think our scaleup network is probably the place that our systems are the most different where Google uses the 3D Taurus with the optical circuit switches to to configure around bad things. We take a much more conventional networking approach where we basically build our scaleup network with a clone network but with a proprietary very low latency link to get around a lot of the overhead of of conventional Ethernet and then have a scale out network that is conventional Ethernet.

25:11 >> Yeah. And just following up on one of your points, I think a lot of the startups don't have that system experience. You know, we Louise and others wrote, you know, the data center is a computer >> and you know, like learning about systems the hard way by building those supercomputers. I think a lot of those lessons aren't obvious until you've experienced the pain yourself. And I think that gives both of us an advantage over startups.

25:45 >> Yeah. He was talking about Luis Barroso and Ers Hosel and Partha book the warehouses the computer that you the something like that but anyways it's in several editions and all the >> warehouse computing >> warehouse computing all the lessons have been learned over the years about building at this giant scale. >> Yeah. So it sounds like the answer to the question was kind yeah it kind of is a mode if you don't the chips there's a lot more to it than chips. and so how much of Nvidia's and Google's competitive advantage today is really about the hardware versus basically software the compiler libraries and how does that work within your organizations and putting it differently if you gave the startup the net list and maybe the layout but not the stack could they could they compete and I more tag you >> Well in the design of TPUs we tried to follow the advice of my thesis advisor >> which >> well that's well that's a wonderful idea. I think that should be a requirement but what was his advice?

26:57 >> don't put off until runtime what you can do at compile time. >> Okay. he so we we tried to do that and we designed the architecture you know in a room of 10 people and two of them were part of the compiler team. So we tried to make a machine that was easy to compile to and so I I think that provided a lot of benefits and also we we've kept the same general architecture and you know this for example the sizes of the memories can increase or decrease and things like that but you know that's like having a PC you can put more dims in your PC and still run, you know, word. So, it's pretty flexible.

27:49 >> What What about the libraries? >> Yeah, there's a lot of frameworks and they're evolving. so, it's important to keep up with those and to have good support. And then also some people like writing their own kernels and so you need to support those as well. How about how about how about you Bill about hardware and software? >> Yeah. So you know the the product is the whole system which includes you know the hardware and and the software and and the software is a really integral part of it but in many ways for deep learning it's an easier problem than the problem we had with general GPU computing that sort of led up to that. you know when we sort of you know you know kicked off you know GPU computing with CUDA and and G80 being launched in 2006 there were literally you know thousands of applications that would benefit from running in parallel but had serial codes many of them written in forran and it was a huge burden and there there were you know you know 100,000 million line codes to port that stuff over and so we made a huge investment in in the CUDA ecosystem the CUDA language itself and you know many libraries underneath you know with to do FFTs well to do matrix operations well and all of this and so when we started you know our first effort in in deep learning in in in 2010 developing the software in a collaborative project with Andrew Wing at at Stanford that turned into QDNN it was a you know wow this is a really tiny application there's you know only a couple kernels that are really important in here we make those run fast this whole thing runs fast that was such a breath of fresh air after looking at a weather code with a million lines of code that that it was it's it's an easy problem compared to what the problem with supercomputing and trying to paralyze those those old dusty decks was. that you know that became easy.

29:41 It also became very critical. We could see that we'd get a program you we'd get one of these deep learning models up and running and then you know over the next six months we double its performance because there are easy ways to lose performance in there. So we started developing tools that would profile and tools that would help us you know fuse kernels and and tune things to make it work well. So I think you know the software is a very very critical part. I don't know how much of a a barrier that's going to be to these startups these days because right now you just sort of tell claude I want a software stack and go away for a week. That that's as you that's my next question.

30:13 As you know, anybody who's knows a programmer has heard from their spouse of shouts in the middle of the night. Oh my god, look what it did. But yeah, we're we're seeing this remarkable event about coding being done by machines rather than by people. So yeah, that was kind of the question. Do you think if there was a remote software is is that about to go away with these the advances in AI or know that's you know that's a polyianish view of what's going on here?

30:48 >> I I think you know any any software moat I think there's still an advantage to having libraries that have been tuned and honed over the years that are easy for people to apply. But I think that moat has been significantly degraded because if you have the spec of that library, I think turning, you know, cloud code loose and and recreating it is not that difficult a thing to do. >> Yeah, I I I agree. The the the moat is getting shallower.

31:17 >> It's getting filled up with sand is starting to be able to walk across. >> Okay. you know, Bill brought this up, but we and when Bill gives a talk, he gives credit to this. We've squeezed a lot of speed of the first years by reducing precision. Yet, like for me, I shock figured out our exponent is bigger than the fraction, right? That that was that's a new thing, relatively new idea. but we can only go so low.

31:44 What happens for generation over generation improvements when we can't shave bits off anymore? >> I can't remember. >> Okay, >> Bill, you're up. >> Yeah. So we we've we've pretty much hit 2x per year over the past 14 years starting with Kepler in 2012 which is the first generation since we've really looked at this as a serious application. and only 3x of that has come from process technology. This is in contrast to you know the microprocessor hey of the 90s when it all came from process technology. and you know I think you know at Nvidia we've kind of led the way with numericics you know going from starting out with FP32 because that's what we had in in Kepwar and and you know you know shipping NVFP before and there's some new things in in Vera Rubin. and you know clearly there's a couple more turns there but we're getting near the end on on numerical precision. There's still a couple clever things that can be done, but there are a lot of other axes that we can continue to to innovate along sparity, circuits, locality, even better models that wind up giving you sort of, you know, more tokens per per watt. but we are sort of getting getting down to the, you know, point where all the low hanging fruit has been picked and we need to climb higher up the tree to find those, you know, 2x fruits hanging from the branches. But we we have a bunch of good ideas. I think they can get us through at least the next four or five generations and then I can retire.

33:06 >> Yeah. >> Yeah. I I didn't >> Is that Is that four or five years or >> eight to 10 years? >> Yeah. I didn't want to suggest numeric was the only thing but that was one of one of the >> that was certainly a lowhanging fruit right as >> that was very low hanging and we also had some numeric innovations. So when Jeff was doing those initial AI things, >> this is Jeff Dean. >> Jeff Dean.

33:38 they were taking doing it in FP32 but then storing it by truncating the low order 16 bits. >> And if you hear truncation and you're a numerical analyst, it's kind of like fingernails on a chalkboard. But but it actually worked. so when we developed TPUs we could run programs in BF-16 that had run on CPUs which were being used at the time. BF-16 is the whacked the the chop it e 32-bit one where you >> right >> yeah so we we could run those and get the same results and so that was really powerful because we didn't have to spend a lot of time debugging things you know at the system level why is this model not working things like that so we could verify correct operation and then once we had that we used that for a file before we started adopting the smaller formats like FP8 and FP4.

34:47 >> so I think one of Google's advantages, theoretical advantages at least is Google both have people pushing the state of the AI and building hardware and so theoretically that's a big advantage and then I think I read that Jensen that Google that Nvidia is going to build up in-house expertise. So what do you think how important how big of an advantage is the access to people pushing to the state-of-the-art versus people who are just going to use open models and do that or you know not not have that close relationship for a male?

35:22 I I think Norm I think you're up. yeah, there's you know leaderboards that show you know some of the open models are performing quite well and so there's there's a race for the proprietary models to to stay ahead. I think they'll you know do to the ability to do better tuning of the models for a particular system either a GPU or a TPU there'll always be some advantage to the proprietary versions but >> you know I was thinking more of you know as hardware people could go talk to the ML people about what's happening next and what should we put in our hardware that if you didn't access to those experts. You would have to guess you see that >> to some extent. the problem is it it takes you know two and a half or three years to from an initial idea to have a system that can be manufactured in volume and the ML people come up with a new idea like every 3 months or something.

36:38 >> Okay. All right. you you can't really design for a super specialized you know a super specialized machine for a particular model because it'll be different and like Bill said earlier you just have to do those basic operations and matrix operations vector operations and stuff and do them well. >> Okay. Okay. If it takes so long and they iterate faster, why are you guys getting into building your own models? We've we're not getting into it. We've been doing it for >> Okay. You've been sorry.

37:12 >> Some time. And we we have a lot of internal expertise, but we I think there's a distinction here between, you know, a company like OpenAI who just had a great paper hot chips on their jalapeno processor where they're designing a chip pretty much to run one model. and you know, there therefore they know you know you know what their relative mixes of matrix ops and vector ops and memory bandwidth and and such. and you know there's a if you have to support all the models that are out there there's a there's a very big variety and it shifts that provisioning of especially memory bandwidth versus math in different ways and particularly with the attention the attention mechanisms are starting to get you know pretty interesting. This started happening with deepseek when they came out with the MLA attention and now a lot of the different models have different hybrid attention schemes. So they'll alternate layers of three layers of state space and one layer of of of full n squared attention. A lot of people are doing sparse attention where they'll do a quick filter and then have to pick out the top k things to attend to. And this this variation in attention I think drives a lot of you know different demands on the hardware.

38:18 And so if you're in the position of being we're going to be a silicon supplier, we're going to support all the models. It it requires you to sort of look across all those models and say what is the right hardware to build to make everybody as happy as you can without making anybody really unhappy. And and that's a hard thing to do. you have to decide, you know, you know which ones you're going to emphasize, which ones you're not. And as Norm pointed out, you've got to aim ahead of the duck. because, you know, it's going to be a couple years before this is out. And in the meantime, a lot of clever ideas are gonna people are going to come up with in the models that we haven't seen yet. And so this is I think what good computer architecture is about is figuring out how to make the most people happy and and anticipate what the applications are going to need and and deliver a product that serves that.

39:04 >> Yeah. And talking about models, I think one of the more disruptive things that that's been recently developed is well it was proposed like seven years ago but mixtures of experts because it requires a lot more interconnect bandwidth and if you want to have quick responses for users you really have to push down on the on the latency. So training is is somewhat easier in in that it's more of a bandwidth thing. Latency is harder to get in computer systems than low latency than higher bandwidth.

39:46 >> Yeah. >> All right. So that's what drove the boardfly configuration. Exactly. Latest TPU. >> Yeah. >> Yes. Given we're at the computer history museum, I could I was struck at the at the hot chips conference. It felt like a recreation of the battles from the 1980s. So in the 1980s when risk risk process risk processors were getting popular people were using myips instruction millions of instructions per second as the metric and but that was up to you to how to define it what you ran all that stuff. So the risk companies were saying well you know you can trust us our but all those other other bastards are lying right.

40:25 So at at at hot chips they were using tokens per second as the metric and what were they running? How big was the model? Nope, it was just tokens per second. So there was an there was an attempt to try and and the solution for the risk one led to the peak companies getting together collaborating on benchmarks which were called spec which were pretty you know there was pluses and minuses spec but they rallied around it and then it turned out even if the you were 10% slower you could have other advantages and so that it worked out pretty well.

40:58 So there was an attempt called ML Perf that to anticipate this problem that's been around for many years but I mean it do you and spec ML Perf was not quoted that was not mentioned I think at at the hot chips so do you agree that maybe ML Perf was not a successful Hops as spec and if not why do you think it didn't and I think it's your turn bill. So, so I I like MLPF because it sort of, you know, gave a a way of, you know, cutting through it did what benchmarks are supposed to be. It was a very nice level playing field to cut through the BS and say, "Okay, you know, how how well do you really perform on on this particular application, but I think it had two issues, which is why you didn't hear much about it at Hot Chips."

41:45 the first is almost nobody submitted to it. And so, >> well, that's a problem. >> Instead, because it was it was a lot of work and we we we did every time. In fact, we did all the benchmarks every generation. >> and every every iteration of MLF would have on 7.1 or something now or whatever. but but a lot of startups would say, "Oh, no, that's inconvenient. And if we actually played by all the rules, we wouldn't look that good. So, we're going to cherrypick this one result and put that on our slides and and talk about that." And so, that's one reason why I think it hasn't really taken off. The other reason is today what everybody is really worried about is tokens per watt or tokens per dollar.

42:21 And there's really, you know, most of the MLF benchmarks don't really address that. And even I think the one that that comes close, it doesn't address it in quite the right way. And so what everybody is showing and actually you did see some slides of this at hot chips is the semi-analysis inference Xbenchmark because it it hits exactly the data that people want and they do it in in a very reproducible way. to get a data to get a data point on that semi- analysis chart you have to have a whether it's GitHub or hugging face a repository of the code that you can load and run and and show on that hardware with this model this is is the tokens per second tokens >> is this something you run yourself or they do it for you >> they they do it yeah >> and they run it on your hardware and give you a number is that how it works >> okay >> I don't remember the details of that but it it's it's >> the seems to be one that's catching on >> I think people people are using inference text more than MLF today.

43:14 >> Yeah. >> Yeah. That ML Perf does require like a full-time team >> and not not an ins and not a small team either. So >> yeah, that's one of the issues with it. >> So it ended up just being too intensive to run like like the the old the old TPC benchmarks were a big effort for these database companies. So expensive and and this alternative was working pretty well. It was both expensive and it didn't really give you the number you wanted.

43:45 >> It It was okay. Other than that, it was great. >> All right. I I I'm supposed to warn the AV people to to we'll do a poll after this. You'll show the poll results of my question after this question. Okay. so the and this is a more open-ended the silicon gold rush has had significant has significant economic societal and even geopet geopolitical implications beyond the technology itself. What do you believe are the most profound impacts of this rapid acceleration in AI hardware development and what responsibility do hardware architects like us and leaders of these companies bear in shaping that future responsibly?

44:27 Bill, I think it's your turn. Okay. So I I wouldn't call it the silicon gold rush. I'd call it the AI gold rush because it's really Russian AI and I think it's really benefiting almost all aspects of our lives. So if you look at medicine, I think you know this is where one of its early applications was in image analysis. But now it's also being applied to diagnosis where it's helping doctors give more accurate diagnosis. And we can have personal health coaches that sort of you know look at us and say I don't think you should eat that piece of cake. You're over your calorie intake today.

44:56 >> that takes a >> and and what if you just ignored that coach? >> Yeah, but that's up to you. But at least he gave you the advice. You know, in education, I'm I'm, you know, a, you know, recovering educator. you know, every student could have a personalized tutor that really understands, you know, what motivates them, how they learn, and and presents the material in a way that gets them through the, you know, difficulties of of grasping concept complex concepts.

45:22 in engineering, AI is already automating a lot of our tasks. I recently needed to design a piece of hardware. I basically wrote the spec and set cloud loose and after fixing a couple errors in my spec, it it came out with a a pretty pretty good design. >> Did it fix this? >> No, it produced exactly what I asked for, which was not the right thing. You have to learn how to write write a good spec, but it's basically enabling, you know, engineers in all sorts of fields to move up the ladder, right? They're no longer doing the, you know, you know, junior level calculations and stuff. They're deciding what needs to get built and they're, you know, managing a team of agent minions that, you know, actually carry out the work and a lot of business processes are are moving in the same way. So, we're all becoming much more productive.

46:04 So, I think in entertainment, it's helping us produce really, you know, wonderful compositions, whether it's in, you know, movies or games or or or music. and so I think there's tremendous number of benefits about AI. But on the flip side, you know, any great technology can be used for good and can be used used for evil. And so, you know, the most obvious ones with AI are deep fakes. And I think we need to move very rapidly to, you know, embrace, you know, provenence and authentication.

46:30 So that basically unless you see an image as being, you know, authenticated, you know, appropriately, you just assume, that it's a fake. I think that there's, you know, danger with, you know, various nefarious uses. Cyber has gotten a lot of attention lately. Although even though people like to talk about the AI side of cyber, the vast majority, I think it's like over 80% of all successful cyber attacks are are human engineering. They're fishing. Yeah, of course. A AI makes that better. you can use AI for pathogen design. and then there's the big you the big risk is as we make everybody more productive that the the nature of employment is going to change and and we'll need more people in certain jobs and fewer people in other jobs. So we need to find some way of of easing that transition for the people who are >> dropping out of one and moving into the other.

47:20 >> Yeah. Help. How do we reskill people whose jobs change? >> Normally you got a take on this? yeah, I think one of the biggest impacts is going to be in science. we they've made a lot of like super impressive results which don't usually make the popular press. So at Google deep mind they folded all the proteins known to mankind >> and got a Nobel Prize. >> Got a Nobel Okay. Yeah.

47:53 and there's there's many other examples in in science. So another thing is Google is using it in virtually all of their applications but it's it's very subtle and in in many places you don't really notice it. So for example, one of the first applications was Google maps. that was when Louise was running maps. And you know the original Google maps would say you know drive you know 1,000 ft and turn right on Tennyson Street right and by integrating the street view database now it says things like you know drive 1,000 ft and turn right at the Shell gas station and that es and you know that's much more accessible to people and especially at night when the street signs aren't lit. You know, Sunnyale has nice lit up ones, but Palo Alto, we don't. So, so I there's a lot of subtle things like that which I think people won't notice, but you know, we'll just take for granted after a while.

49:13 Okay, let's see. Can you show the poll? Did that work? Okay, so here's my question. the biggest bottleneck to continued widespread deployment of AI chips is energy availability, semig manufacturing capacity, cooling of hot chips, memory capacity, bandwidth, cost of AI chips. Don't know. >> More people don't know. Okay. Well, I guess Oh, this is >> dynamic. We we we won't >> we it's still changing, but clearly right now it's energy availability. Well, maybe I should just maybe I'll give it 30 seconds.

50:00 >> So, what what what you guys could answer this question yourselves and you you could disagree with the audience. What what what do you think about what's this biggest limiter of those those things? Well, I was just going to talk about the energy availability. >> Okay, go ahead. >> so one of the issues is, you know, people are talking about gigawatt data centers or 5 gawatt data centers and the transmission lines are very hard to get permits for and communities don't like to have, you know, big power lines running over their houses and stuff. And so a a lot of the hyperscalers and other companies are moving to a model where they're doing on-site power generation. And you can do that in various ways. Some companies use natural gas to generate electricity. But if you use if you put it in the right place, you can get most of the power from wind and solar. And so that's the approach that we're trying to take >> which is car carbon-f free energy and will not will not you know add to the the the load of the on the climate >> right >> but by going off the grid and using carbon free resources you can build data centers without clobbering things.

51:25 >> Yeah. you have an issue that you need steady power for your data center and you know the sun is only out during the day and when there are no clouds and the wind only works when the wind's blowing but u you know in general we've seen the various sources of supply being reasonably well balanced I think it's just the natural market forces that are deploying resources to do that but land power shell which is sort of the three things that you need you know to build a data center tends pot tent to drive that demand and you know while a lot of people are are looking at natural gas and colllocating generators. I think you know gas turbines are sold out for like the next five years because of of this demand and there are companies that are taking engines off of airplanes and converting them to to be used as as generators for stranding lots of passengers somewhere.

52:16 >> Yeah. But but if that's what we do, if we if we go to natural gas, we're going to have a giant carbon footprint. if if we don't >> Yeah, but it's often used to back some more renewable source. They'll have they'll have solar and wind, but because you know you can't count on that and we don't have the storage capacity at this point, although there are some very very good technologies that are >> and so you think like lithium batteries aren't going to be a >> I I think lithium batteries are too expensive per kilowatt hour to even to ride through the kind of 20 to 100 hour thing you really need to bridge a gap when you might not have you know sunny days for a period of time. There are thermal batteries that are looking very promising and there are people starting to build data centers with those that are cheap enough per per kilowatt hour that they could fill that gap.

53:03 >> So if you're doing serving to consumers though, >> yeah, >> most consumers I mean there's a dial cycle, right? And so you don't have to have full power at 3:00 a.m. when you're serving. So >> it has to do with with keeping use of a capital resource, right? you spent, you know, $10 billion building a data center and you'd like to have that capital resource be busy around the clock. And so if the consumers here are all sleeping, you'll sell those GPU cycles to the consumers on the other side of the world where it's >> where it's the latency is too large.

53:35 >> Yeah, it depends on the use. Yeah, >> we like we like quick responses for our users. >> What what about nuclear is also carbon free and there's these small modular reactors. I think Google's certainly made an announcement in this space. Do you guys have thoughts about nuclear as to help data centers? >> Yeah, we we're, you know, you we're looking at all all technologies. We actually have a guy whose job it is to sort of keep track of energy technologies and and advise on on on data centers. and it it looks promising, but you know, it's always been expensive compared to, you know, if you look at the what they call they call the levelized cost per kilowatt hour. you know compared to to natural gas. I mean natural gas is is very tough to beat and even if you're worried about you know the the carbon natural gas with carbon sequestration I think still winds up beating nuclear on you know levelized cost per kilowatt hour.

54:33 >> I I'm a big fan of pumped hydro but >> in Illinois where I'm from it's not too practical. >> Okay. now I'm going to go to audience questions just about when I said I would to what are the biggest ch and the way the audience audience questions get voted up and down. So this is the most popular one. What are the biggest challenges to resolving the known supply chain barriers associated with AI hardware which I presume is the over demand compared to the supply. So do you have a take on that the biggest or what's the biggest challen I guess what's the biggest challenges?

55:12 I mean the biggest challenge is it takes a long time to build a fab. >> Yeah. >> and and I think you know right now the the place that's being felt the most painful is on the memory side and and the memory manufacturers are loving it because they can you know charge many times more for the same part if they'd had them in plentiful supply. So they maybe they're less motivated to build the fabs than they otherwise would be.

55:34 >> Yeah. but I think it's it's it's always this thing where you're trying to project demand. Then when that demand exceeds what you projected, you know, you you've got this two or three year delay to actually build the production capacity. >> If if I had been paying more attention, I probably could have figured this out before it happened to some degree because memory densities DRAM scaling has stopped basically. And so since computers keep needing more and more memory, even ignoring the need for, you know, memory in AI, I I I think it's finally tipped the balance where for quite a while they there'll be it'll be a better business. It used to be one of the worst businesses DRAM manufacturing.

56:21 >> So yeah, basic you've already you've taught me that SRAMM is already plateaued. >> Yeah. And but you you DRAMM has basically plateaued but the demand hasn't plateaued, right? And so it's going to be a long time. Yeah. And I I think it'll be interesting for the many companies because their history is over overbuild fabs and then the prices go down. They've got de a couple of decades of that. And so are they just going to say we're highly profitable. What's wrong with the way things are today? And just >> fortunately competition hopefully will fix that and they'll we're highly profitable. we'd like a little bit more share and so they'll build their fab than the other guys will too and >> hopefully it will >> yeah but fabs are expensive and as you pointed out take a long time not only to build but to get the yield up and all the other things >> I'm looking forward to what you guys are going to say publicly about this next question how significant will the impact of quantum computing be on the topics discussed this evening when the technology matures >> I would say very very little so I think quantum computing is a >> so glad you're here.

57:26 >> Is a great technology. >> Bill Deli said that >> not Dave Patterson. Okay. >> You get beat up by Benson tomorrow. >> Yeah. So very little go >> and but the the reason is there are sort of two really good quantum algorithms, right? One is to use quantum computing to simulate quantum chemistry and the other is Shor's algorithm which is to use quantum computing to basically factor the product of two large prime numbers and break a lot of modern cryptography. There's there's a third algorithm, Grover's algorithm, but for optimization, but it only gives you quadratic speed up, not exponential speed up. But all of these algorithms have the property that they take advantage of the fact that quantum computers are large computation, small data machines. In people's wildest dreams, they'd like to get a few thousand error corrected cubits. and then you get from the exponential speed up of superp position, you get to do tremendous amounts of calculations on a few thousand bits. AI is the other way around. It's a large data, relatively small computation problem. So quantum computing is not going to have a measurable impact on either training or inference of for AI.

58:32 >> I I I know who I'm going to call on questions of the future. Yeah, it's I mean the AI re there we this we were 70 years into AI, right? And there was a lot of ideas that didn't work, right? It's been learn from data, the big data machine learning thing. That's the breakthrough and that's not like you said. So >> the area where it will have an effect is not on computation I don't think but communication so the last touring award or maybe the one before it I guess what >> Nobel touring award for quantum. Oh yeah quantum for training award. You're right. You're right.

59:11 >> Yeah. u quantum transmission and cryptography. But I thought quantum transmission was very low latency but not much data. Is that right? >> okay. That we're out of our we're not you don't I mean if you're sending a 300 bit long key you you don't need high bandwidth. >> Yeah. >> And then you can send all the rest of it >> encoded and then it doesn't matter.

59:42 >> Your keys need to be longer than 300 bits. >> Yeah. this may be outside of our expertise but that's the next question. What are the national security implications associated with AI hardware maturation? I guess if you've worked in stuff like that, Bill, do you have any >> I'm not sure I understand the question. >> Okay. I think I don't know that one either. this is a specific company. I don't know if you're going to What is your take on etched architecture?

60:14 essentially etching an LLM on the chip or so so like Talis or something like that. >> Yeah. Well, Talis would be the extreme where they're planning to use ROM. >> so, these these architectures are less programmable and they're kind of building the model into the hardware, right? Well, my my understanding is that that's and >> people can correct me if I'm wrong, but etched the company started out with that thesis and then it became more and more programmable and now it's >> is it they try to make a product?

60:47 >> Yeah. >> Yeah. Well, I I thought I read something in the popular press that Google was building some ship that had >> Would you like to reveal any secrets? >> There's research into everything. >> Oh, that's a great answer. But but I think it's it's interesting because if you if you take the talis approach if you say you know they're they're going to the extreme not even that we're going to take a particular model and totally optimized for that but we're going to take the set of weights and burn them into into ROM because that you know will wind up being you know maybe four times as dense as SRAMM and way less dense than DRAM by the way. you've got to have figured that there's some model that is not changing very fast because right now you know the frontier models are coming out with a new release every month or so.

61:36 and even if you look at the various open source models there's a clever new idea every couple weeks that you know would completely break even trying to you know make it too closely matched to the model even if you made the weights programmable. >> I think this is we're in the early days still and programmability is critically important. >> Yeah. I mean I think this is what kind of what you said. This is a completely different universe from CPUs with million line legacy code and millions of programmers. It's really small amount of code and not as many there and there's tremendous pressure to innovate and so that's allowing this that makes it easier not only for the hardware people it makes easier for the software people or the algorithm people to innovate and deploy it. So it's it's a remarkable rate of change and this idea that you could pick a model and get a chip out in time and how long is that lifetime and stuff. It's an interesting bet but you know it'll be interesting to watch how it works.

62:35 Okay. what is the right balance between centralized and distributed compute? What are the implications for edge compute hardware? Well maybe we could just change that. You what do you get? You know, we all think about the data center. >> Well, you know, we we ship an awful lot of products that go into autonomous vehicles. Every Mercedes above a certain level has an NVIDIA processor and NVIDIA software in it for its self-driving features. And and that's the example of if you can put it in the data center, you do because it's much more economical to deliver compute in a data center, but for reasons of latency or you you have to be able to operate reliably with a network partition, you may need to compute on the edge or in some cases you're acquiring lots and lots of data, you can't afford to send it all the way back to the data center, you have to do some production on the edge. But I think the general rule is if you can afford to do it in the data center, if one of those other factors doesn't limit you, you do it there. But if you have to do it on the edge, you do. and and it drives a different set of requirements.

63:32 >> Yeah, it the Google Pixel phones have edge TPU so it handles a lot of stuff that's relatively simple like speech recognition is pretty simple these days compared to these other big models and so but if if there's something that's too big for it then it yeah needs to go over the network if it's available. Do you talk to the people who do the computers for Whimo? Are is that are you familiar with that at all? Is that another part of the company?

64:07 >> There, you know, they set up Google to be part of Alphabet and then Whimo is W. And so we we we don't Yeah. talk that much. >> Okay. since floating point precision achieves comparable AI results, will there be ever be room for analog rather than digital systems? >> Yeah, >> you guys must hear this all the time.

64:40 >> What about analog for you you can do multiply so cooly in analog? Isn't that the future? >> Yeah. So, one of the things that when you actually build real systems for a living, you have to do is test them to see whether they're right or not. And if you have something that's analog or, you know, doesn't get the same result each time, it makes it really hard to test the device to see whether it's working correctly or not. I mean, if if you're probing wafers, you have to send test sequences and then you read back bits at tremendous speed and and they have to match one for one. So, a analog even if it can be more effective in some circumstances, you have this problem that you might not always get the same answer.

65:41 >> And so, how do you know if the if the chip works or not? >> Yeah. Yeah. Yeah, but even if you could test it, I've seen few cases where analog actually has an advantage. We we continuously re-evaluate this because it it's a very seductive, you know, argument that, oh, you know, you you want to do a matrix multiply, we'll just, you know, put, you know, the activations in as you know, current and then you have, you know, a resistor and you think of that as a conductance.

66:06 You know, V is I * G. And you you've gotten your matrix multiply for free. And by the way, you can then sum a lot of those together also for free and you wind up, you know, getting sort of free computation. Well, it's not not free. There's no such thing as as a free lunch or a free multiply. it it costs something, but what's even worse is in analog, you don't really have a reliable way of storing a value to to store it for more, you know, for very short instant of time, you can put the value on a capacitor, but it's going to leak off, especially in modern semiconductor technologies. And if if you need to move it any distance, it's very expensive to try to move it in analog.

66:45 So so typically what people do is they'll build a little array to do you know inmemory compute analog do do a you know matrix multiply of some you know activations times weights and and then you sum the results and get the value out and then you have to do an A tod conversion. And if you look at the the fundamental requirements of energy to do that A toD conversion, it winds up limiting you to something, you know, if you wanted to be at say 8bit precision to something on the order of of five terops per watt. Whereas with conventional digital technology, we've built accelerators that are 100 terops per watt. So you're you're down by more than an order of when you include the convert.

67:21 >> If you if you can do if you can do it without converting, >> you can stay analog, you could potentially make an attractive device, but if you have to convert, you're going to lose. >> Yeah. And I haven't been downstairs to the museum part lately, but you know it starts out with some analog computers and there's a reason why they stopped being used around World War II. So >> this what you know in addition to you know the the excitement around software tools and talking to my younger colleagues a lot of excitement about using AI for hardware design and so this question is what will the next chip design software look like? Will they be radically different from what we have used for the last 30 years? This seems like a the set one of you both of you must have an opinion on this.

68:14 >> Yeah. So I think the chip design software is going to be an LLM that you talk to and >> okay that's you're very quotable that's >> I want this chip >> so we should sell sell sell my ECAD stock tomorrow. >> Well you know I think they they have a lot of expertise and a lot of the pieces that are needed to plug that together. >> Okay. So we need to you need to capture that knowledge in your LLM.

68:37 >> Yeah. And and the LM needs to know how to run the tools, right? If you're going to need to run those tools >> to do you know the the place and route and the verification. But I think you know to me I look at the where we wind up spending a lot of the human time in in designing a chip and it's in verification. >> Yes. >> LLMs are really good at writing tests right you can say make sure this works cover all the edge cases and it will write a whole test suite for you and that's where you know 75% of our labor goes.

69:04 >> Yeah I I was going to say the same thing. It it's that you know the results we presented at hot chip showed you know like 10% improvements in doing the design and stuff but design verification there's it's the biggest part of the team is the DV team and that they need all the help they can >> yeah and we're seeing bigger than 10%. We're seeing, you know, for average engineers a doubling of productivity and for really good engineers 10x.

69:32 >> What I was talking about was >> layout circuit basically performance. >> Oh, okay. Yeah. >> Oh, you're talking about performance, not human productivity. Okay. >> Yes. Yeah. >> So, but you you're talking about for all tasks or for for verification. >> Yeah. I'm trying to remember the benchmark study that was for was for a particular it was for a particular task. you know, the average came out to be about 3x. it made engineers just way more productive.

69:57 So I think that that's going to be the the design tools of the future and and there are a whole bunch of startups now and and again this is the the thing in AI the real the real gold miners are the people attacking a vertical. There are a whole bunch of startups attacking the EDA vertical. >> let me do one more. Yeah I'm sure. What do you think about a CPU GPU ecosystem to make space for compute in memory or memory bound workloads?

70:30 >> Well, what do you think about how about making the CPU closer to the GPU? You must have a comment on that. >> Well, you know, our ours are pretty close, right? So, they they they sit on what we call our, you know, chip to chip NV link. It's actually, you know, technology internal that we call GRS because it's ground referenced. Simply it's a single-ended very high-speed link. And and so you have to ask what do you get by making them close? And and we get a couple things. One is we want to provision that link between the GPU and the CPU so that all of the and I forget what it is you know 1.8 8 terabyte per second of of LPDDR5 bandwidth that's attached to the to the the Vera CPU and and I I don't memorize these numbers that maybe but all of that CPU memory bandwidth can be routed to either of the two GPUs that's attached to it. So the link is provisioned so that link is is never a bottleneck. Then then the question is you know what do you need in terms of latency and because fetching from that memory is necessarily a relatively high latency operation. the little bit that you add with the link winds up not being that material and it's only a single link. You're not going through a switch or anything like that. And then there's control interactions. But if you think about the, you know, it's basically when you launch a a kernel that you need to to do that and and that's, you know, there's enough latency, but the other parts of that that making that link faster isn't that critical. It turns out we have many tens of little risk 5 CPUs on our GPU, but they're all there for little housekeeping operations that the user never sees. you know, for the CPUs that the user programs, you know, the, you know, Envink chip to chip is actually just about as close as you want it and is is kind of the right the right balance.

72:13 How how about for TPUs? >> Yeah. So, I think long term the CPU that's associated with the accelerator is going to be doing mainly housekeeping and like babysitting kind of chores. I think the the real interesting thing that's happening now is a genetic computing. So, if you say, you know, write me a program, it's got to compile it somewhere. And you're not going to compile it with on a matrix multiplier systolic array, right? So you've got to have real CPUs with you know serious memory systems and stuff and serious networking in the same data center and >> right but in the data center it doesn't have to be that close because you're firing off a compile job, >> right? But I think that's that's where the the CPUs really will come in is doing things like compiling.

73:10 >> yeah, it doing the tool calls. >> Yeah, >> it doesn't. Yeah, but if you're going to do compiling cast, I I don't quite see why it it has to be close to do that because that's up to you. >> No, it doesn't have to be close. It's just in the same data center. >> what I was I I I was going to do a closing question, but we we have a suggestion as well. basically about advice to younger people. I was thinking like high school students, college students. The one of the questions we have here, let's see if I can find it.

73:41 I'm 10 years old. What advice do you have to me to prepare for my future? I I I think your kids are all of our kids. I don't think any of us have Well, I have grandkids that old, but I I What would you say to 10 year olds? Do you have any advice 10-year-olds? Then I'll ask you about older students. >> So, you know, study math and science. I think that you need a good a good foundation and and there's no substitute for having a really strong, you know, background in mathematics and and and basic sciences.

74:19 >> How about you, Norm? >> I think another important thing is communication skills are really important. So, learning how to write well, I know, LLMs can do it for you, but I I think there's there's going to be cases where you don't have one handy and you still need to communicate well. >> Yeah. Yeah. Think clearly do that. So, yeah. My last question is going to be given the incredible pace of innovation we're witnessing, what advice would you each of you offer to young computer architects and engineers entering the field today looking to make their mark in the computer architecture space? You guys have certainly made great marks in your careers as you look back you know what what let you make those marks and you know what you think you should offer what advice would you give for this current generation?

75:11 >> Well, I think the world's very different today. I mean I I had great fun early in my career. I designed a machine at Bell Labs, you know, you know, drawing all the circuits with pencil on vellum and having a technician who did the layout, you know, for them and and it actually all worked on first silicon without ever doing a simulation. but you know, having great manual logic design skills, I think it no longer has any value at all.

75:34 >> I I know a lot about punch cards. I I still but but I think I I would have three pieces of advice for an aspiring computer architect. The first is to master a vertical. The second is to be very broad in your understanding of computer technology and then the third is to make AI your partner. And and I think you know the reason for the first is I think a lot of the value today is in understanding what to design not in having great you know manual logic design skills to realize it because the tools will help you know the AI will help you realize it. making AI your your partner. but then I see many computer architects who limit themselves because, you know, they'll ask the memory guy, oh, can we build a memory like this? And the guy will say, no, you can't do that. But if you have a really broad understanding of circuit design, you'll realize, you know, he's not thinking enough out of the box. And you can build a memory that that that does that. This machine at Bell Labs, by the way, had had a 3T DRAM on it.

76:28 >> Wow. >> all all the all the circuit guys told me, "Oh, that's not going to work." Mhm. >> And and I went and I you know prototyped it up and and and convinced myself that it would work. We actually did did a test chip before we did the final chip to make sure it would. But you you had you have to be broad enough in your understanding that you can do circuit design when needed and and things like that to sort of get outside the box and get outside the limits of the conventional thinking of what the toolbox you have available to you. and then I think you know you know people really need to figure out what is a uniquely human part of the human AI partnership in in doing computer architecture and and you know practice at being good at what the human can do because there's no point in doing what the AI can do because it's just going to do it better. But but you have to be able to so far you have to be able to recognize when it's screwed up, right? So yeah.

77:18 >> So do you have to know how to do it manually to be able to recognize when it's >> I can definitely recognize when it's screwing up, but maybe that's because I can do it manually. >> Yeah. Well, yeah. >> The other thing you can do is you can get one AI to check the other one. >> Okay. How about you, Norm? Advice advice to the next generation. >> Yeah. for computer architects, it's got a long history. and you know going back 70 years and or more and there are a lot of lessons learned along the way and I think you know the one of the best things to do is just learn about all those historical lessons because some of them apply today some don't apply today and there's no point like reinventing something that didn't work the last seven or eight times people tried it.

78:17 >> Okay, with that I think we invite Mark back up on the stage. >> Yeah, thank thank you so much Dave Normanville. What a great conversation. Let's give them one more round of applause. And and when you think about it, we are called the Computer History Museum, but our conversations really are about the past, present, and future. And so that's really what the museum is about. We would love to get your feedback on tonight's program. So you can scan that QR code, please tell us. It helps us plan other events. The other thing you can use for a QR code is help us out here. we are reliant on your donations as a museum. A lot of people think that it is Google and Nvidia that is the big companies that support us. It's actually all of you. So, thank you guys who are supporters. If you're not a member, we encourage you to join.

79:14 we'd really appreciate all the help that you can give us. that funding will help us put on our next event which is Boris Churnney from Claude Code. So, it's the flip side of what you guys are talking about. So, if you enjoyed tonight, I think you're really going to hear enjoy hearing Boris speak about Claude Code. So, thank you all for coming. thank you for your questions. Thank you Mark and Mary Stevens for your support tonight. And have a safe travel home. And thank these guys one more time.

79:45 And we got don't forget your mark together without