Section Insights
Introduction and Upcoming Events
What are the key topics and events being discussed?
The host introduces the show, highlights the upcoming Anthropic IPO day live video summit, and outlines the main topics for discussion, including Jensen Huang's statement about Nvidia, customer diversification from Nvidia, OpenAI's interest in Goodfire, and mechanistic interpretability in AI.
- The Anthropic IPO day live video summit is scheduled for November 19th.
- Jensen Huang expressed willingness to return as CEO of Nvidia.
- There is growing interest in AMD chips as alternatives to Nvidia.
- The show will cover significant developments in AI research and industry.
AMD's Growing Market Presence
How is AMD positioning itself against Nvidia in the AI chip market?
AMD is gaining traction among customers, with increased spending on their AI chips and a growing number of cloud providers offering AMD technology. Customers are attracted to AMD for better pricing and performance in specific AI tasks.
- AMD is seen as a viable alternative to Nvidia, gaining more customers.
- Cloud providers report increased interest in AMD chips.
- AMD chips are cheaper and offer better margins for certain AI applications.
- Nvidia remains a leader, but AMD is improving its technology.
Challenges in Accessing Compute Resources
Why are companies moving towards building their own data centers?
Companies are increasingly taking a hands-on approach to data centers due to difficulties in obtaining necessary compute resources from cloud providers, who are often reluctant to sell to potential competitors.
- Some companies are building their own data centers to secure compute resources.
- Cloud providers are hesitant to sell to competitors, limiting access.
- The demand for compute resources is outpacing supply, especially for inference tasks.
- Smaller cloud providers prefer selling to software companies rather than resellers.
Understanding Mechanistic Interpretability
What is mechanistic interpretability and why is it challenging?
Mechanistic interpretability involves understanding AI models based on their behavior and outputs, but it is complicated by the notion that models can be seen as 'grown' rather than 'built,' leading to difficulties in comprehending their internal workings.
- Mechanistic interpretability focuses on the behavior of AI models.
- AI models are often perceived as complex and alien, complicating understanding.
- Researchers struggle to interpret models despite having access to their outputs.
- The concept of models being 'grown' highlights the challenges in AI interpretability.
OpenAI's Collaboration with Goodfire
What is the nature of OpenAI's partnership with Goodfire?
Goodfire will work on-site at OpenAI to assess and ensure the alignment of AI models with intended goals, addressing concerns about misalignment and enhancing interpretability efforts.
- Goodfire will help OpenAI monitor model alignment during training.
- OpenAI has been criticized for its interpretability efforts compared to competitors.
- The partnership aims to strengthen OpenAI's interpretability research.
- There is a growing internal enthusiasm at OpenAI for enhancing interpretability.
Transcript
0:00 Heat. Heat.
0:37 >> >> Heat. Heat. >> >> Welcome everyone to the information ti.
1:15 My name is Akos Pasricha. It is Tuesday, November 17th. A reminder before we get started, we have our Anthropic IPO day live video summit on Thursday starting at 11:30 a.m. Pacific, 2:30 p.m. Eastern. It is available for all subscribers to the information. We're going to talk through the market's reaction in real time as the stock starts trading. We'll talk about what it means for the IPO market, how Anthropics Tech currently stacks up. If you are a subscriber, you can register register at the information.com/events.
1:48 If you are not yet a subscriber, I encourage you to check out our website. Today on the show, Jensen Hong issued his first statement since stepping down as CEO of Nvidia, saying he would be honored to return as CEO of the company if the board makes that decision. We're going to unpack that statement. We are also going to talk through our exclusive reporting on the extent to which customers are looking to diversify their chips away from Nvidia. We also have new reporting on OpenAI's interest in the AI startup Goodfire and the latest craze in AI research, Mechanistic Interpretability. We're going to break down that term for you and we are going to close out the show with a conversation with the CEO of Foundry Robotics. It's going to be a good one.
2:32 So, let's get right on into it. Former Nvidia CEO Jensen Huang issued a statement this morning saying that he would be open to returning as CEO of the company. I want to bring on Martin Piers, editor at large for the information to break it all down. Martin, welcome back to the show. It's great to have you here. >> Akos, how are you? >> I'm good. we got the statement here from >> It is really huge news, isn't it?
2:59 Chancell wants to come back or he's willing to come back. Well, of course he's willing to come back. He didn't want to leave. This is a the statement of the obvious. All he's trying to do is to, you know, u sort of drum up more support for his return. But the problem we have is that we don't know why the board pushed him out last week. and without knowing that, it's very hard to assess whether or not the board will be so stupid as to change its mind so fast and bring him back.
3:35 >> okay. Presumably the board was aware of something that we don't know or they were more concerned about the export stuff than they have indicated. the fact that they last night announced that they'd formed a new committee to oversee exports indicates that that is an issue. There's been a huge amount of reporting that Invidia might have helped along the smuggling of chips into China. So you know you would imagine that is what they are concerned about and if that's the issue I don't see how they can bring him back. But as I said, >> because they never announced why he left in the first place, >> we just don't know.
4:19 >> Right. So, look, I mean, you you have these loud voices out there. You've got these petitions from Silicon Valley elite. You have people who have stakes in Nvidia that are that are leading the charge on some of this >> very small stakes. >> Small stakes. So, so this is what I want to get into which is that if you look at the shareholders of Nvidia and you know what power the shareholders could conceivably have over this situation. I mean this is what you wrote last night in the briefing. I mean what what could the shareholders do to to move this situation along somehow?
4:53 >> Well yeah I mean they do have some power. So, Nvidia's bylaws have a provision that was introduced in 2024 that allows a group of hold of investors who have held 15% of the stock between them for more than a year to call a special meeting of of investors and to presumably they could replace the board. Now, the thing is getting 15% is not going to be that easy. The biggest holders other than Jensen who only owns 3.6% the biggest investors are Black Rockck and Fidelity and Vanguard. people like that. Those kind of institutional investors don't typically take action to overthrow a board. So it's unlikely that they would participate. And but you know, of course, who knows? We can't rule anything out. But that is something that if there was enough investors who'd been long-term holders, if they got together, they could try and do that. So, there's always that possibility. And >> and and what do you think? Do you think that that they should replace the board, change people on the board? What what's your view?
6:04 >> No, I don't think that they should replace the board. I think the board should ignore all of this and maybe the board should explain more clearly why he had to go so fast because you know they had started this investigation they said at the time they had confidence in him and then days after that he's out and they didn't explain what happened. So I think what would the board you know ought to do is to issue some kind of statement that gives us some understanding of why they changed their mind so fast. Now people have told me they can't do that. They can't admit anything. Maybe they can't go into too much detail but they probably could I think explain something of their thinking so that these investors who are saying like you know he deserve more of a process which I think is true. I mean, Jensen deserved something. he didn't deserve to be pushed out that quickly, but we don't know what the board is aware of. That is the problem.
7:06 and so I think the board should explain its thinking to the extent it's able to. And once we understand that, then we can, >> you know, then we'll be much better informed. And and and last question for you, Martin. I mean, like what an explanation like this, I mean, how would this how would this come out? This would be like a like a special statement from the board. This would be a filing like what what would what would this be?
7:32 >> In Nvidia has earnings tonight. >> Okay. >> Why couldn't the chairman of the board go onto the earnings call and say, "Look, there's been all this drama about what we did. We just like to explain they could give some some idea of what they are thinking that they didn't come to this in as much haste as it seems that they did. They had good reason to do it. They cannot bring him back for the same reason that they pushed him out. because if they don't do that then there's just this uncertainty about what happened. That's just going to hang over the entire situation. So, so what you're suggesting here is that an earnings call could become mustwatch television.
8:18 >> Well, look, that's me. I mean, I'm sure they'd say for all sorts of reasons, they can't do it that way. >> but I don't know. You know, surely some kind of public statement explaining their thinking seems to be called for at this point. >> Great. Okay. Well, Martin, I want to thank you for coming on. That is Martin Piers, our editor at large here at the information. With all that's happened at Nvidia, a big question for the company is how long it can keep its clasp on customers. New reporting from my colleagues Katherine Plloff and Brian Barwick suggests that customers may be looking elsewhere, but this actually began long before Jensen left. I want to bring on Katherine Pearlloff to share more of her reporting. Katherine, welcome back to the show. It's great to have you here.
9:03 >> Hey, Akash. Okay, so AMD is widely seen as the closest competitor to Nvidia. where is their technology at? I mean, are customers telling you that they are absolutely choosing AMD over Nvidia? >> I don't think it's it's that cut and dry, but I think AMD is definitely getting more customers. People are spending more money on renting AMD's AI chips. and at times using using it instead of Nvidia. so some data points here we talked to several cloud providers who said that you know who offer AMD chips who said that their the traction and the interest they've gotten from customers from AM for AMD has really increased this year. we talked about a couple actually kind of AMD specific NeoClouds that AMD has invested in who have really seen their revenue you know take off and really grow in the past year. Those are TensorWave and Scyorium. and you know we've also you know a couple what one you know we've also heard some startups are finding that AMD kind of gives them better margins you know for certain AI tasks >> because they're they're cheaper.
10:24 >> They're cheaper yeah they're cheaper and they also have a lot of memory which makes them really good for inference. so you know, I mean, I think Nvidia is still seen as bestin-class. but AMD getting better at doing more things and if there's if there's an opportunity you know, customers are looking to them, you know, not to mention that, you know, it's still kind of sometimes hard to find Nvidia chips and they can be really expensive. So customers are looking for alternatives >> and it's not just AMD, right? I mean this is also traction with the TPUs that you wrote about.
11:02 >> Yeah, TPUs are also getting more traction for the first time Google has started selling them kind of very nently but outside their own cloud and you know we talked to one of the early cloud providers that's providing TPU ships outside of Google. It's a small company called Cirruscale and they have seen like hundreds of millions of dollars in interest for TPUs. and you know my colleague Ryan Barwick spoke to some startups who were at times swapping out GPUs say within Google cloud for using TPUs. So I think it's definitely getting traction you know with >> what about what about tranium? What about trrenium from Amazon?
11:44 >> Trannium that you know this year developers have also found that tranium is easier to use and have been testing it more. so I think you know all of this I think trading was probably a little bit behind TPUs and AMD. they, you know, you can only get them in AWS, which maybe is fine. AWS is the biggest cloud provider. but yeah, I think this year, especially, as more companies are doing AI, more companies are running AI inference workloads, Nvidia chips still are expensive and hard to come by. You know, it's really been the year where companies start looking at alternatives and earnestly testing them. So you talked about AMD being having a higher memory capacity and then also being cheaper and you know as we think about the advantages that these chips have potentially over GPUs. I mean one of the moes that Nvidia really had was around their CUDA software that everyone raved about these other chips AMD t Google's TPUs and then and then tranium.
12:48 Is there a competitive software suite at all around these competitive products that is getting any traction against CUDA? >> I think that it's getting better. So, you know, one example I talk about in this article is that TensorWave, this AMD specific cloud, they used to have spent a ton of time with their customers sort of configuring the AI models to work on AMD chips. and now in the past year or so the time that you know their staff have to spend with customers getting the chip the AI models to work on the AMD chips has decreased by 80 to 90% because the software is more user friendly for their end users to just sort of plug this in themselves. So I think things are getting better. I mean and same with Trinium. they made improvements to their software in the past year, notably kind of making it more compatible with PyTorch, which is a a big tool developers use when configuring models to chips. but you know, I mean CUDA is still definitely more wellknown and there's more sort of kind of open- source and developer resources to kind of figure it out if you get into problems, >> right? So, okay, let's talk about then how the story may have changed in the last week then in a more acute sense because one of the things you you point out in the story is that this is this has been going on for a couple weeks, couple months now, this is not like anything has changed in the last couple days, but I am curious in the past couple of days these customers that you've talked to, have they had any kind of a business reaction to Jensen leaving? Have have they have they said anything about hey we're actually concerned about innovation at NVIDIA now and we we've started to look elsewhere.
14:32 >> Well, you know, I talked to one company Super Language as an inference provider. and their CEO said, "Listen, we're still in an Nvidia crunch and to resolve that crunch, we need to spend, you know, Nvidia needs to spend money to increase capacity. And if you've got the CFO now at the helm, maybe they're not going to spend as much money as they need to increase capacity. And so, I'm not going to be able to find Nvidia chips anytime soon.
14:59 So, that's why I need to you know, more seriously look at competitors." having said that, I will say that like, you know, one thing that complicates this whole narrative is that Nvidia sells a really good relationship with TSMC and they can get priority for manufacturing capacity. So, I did speak to some customers who were interested in testing AMD, but when they asked to use it, they were told, hey, you know, it might take even longer to get an AMD to rent an AMD chip than it would to rent an Nvidia chip. because even though Nvidia is more popular, hard to come by, sometimes the supply for Nvidia is bigger just because they've been able to produce more of their chips with TSMC.
15:37 So, I think yeah, I think sort of to the effect of all the drama of last week. I think the question is like how is this going to sort of affect availability of Nvidia chips, that's really what's on the mind for customers. But I think, you know, to be fair, like they already have a really good supply chain locked in that can't easily change just with like, you know, an executive leaving, >> right? I want to pivot to a related story that you wrote last week and and it's related because you you write about another group of customers here that could be coming online in the in the the chip ecosystem. And these are these inference startups, companies like Fireworks, Base 10, Together.
16:24 you know, we we we talk about them time and time again on the show and I I think I I ask people every time to to explain it concisely what they do. I mean, the way I understand these are companies that they they rent compute out to customers and then they also rent access to models. They kind of bring everything together. Is that the best way to think about it? >> Right. They're sort of a I mean they see themselves as software companies that help optimize running AI models and they make it easier to run different models particularly open- source models on the cloud. But I mean right the key distinction why they're not cloud providers is they traditionally have not actually owned the infrastructure you need to run AI models. They haven't owned the chips. They don't rent data center space. They basically buy compute from cloud providers and then sort of resell it with their software on top of it.
17:13 >> They they rent they rent they rent the compute from the cloud providers and then re rent it essentially. >> Right. Right. >> Yeah. And I guess Yeah. >> Well, okay. So now tell us where the story is going because I mean they're buying chips according to your reporting, >> right? I mean the what I was going to say is like is that a sustainable business model is the question of my story because you know you're kind of right like and to what extent are you sort of marking up something that's in short supply. So yeah, we have we reported that basically these companies are working to become more vertically integrated aka buying their own GPUs.
17:47 not necessarily you know building their own data centers but some have you know kind of taken more of a hands-on approach. They're like together is more kind of involved in the data center process but you know buying their own hardware so that they can kind of make sure that they can have access. And part of the reason they've taken this step is that it's been harder for them to get the compute they need from cloud providers and they've had to start working with like a ton of them just to get all >> harder to get because the cloud providers won't sell it to them.
18:17 >> So there's there's a couple reasons. One is absolutely that so some of the cloud providers are I mean some of them don't want to sell it to them. We have this example in the story with Coreweave was sort of you know reluctant to keep selling to base 10 because they're like we just got into the inference game ourselves. So Corbus has their own inference business where they sort of have this sort of higher level software that lets their customers run AI models and why would we sell our scarce compute to a competitor when we could just use it ourselves. So that's and you know we also talked to Digital Ocean and other smaller cloud provider who said we rather sell our compute to software companies than resellers. they didn't say inference companies for resellers but you know I think that there you know cloud providers are really over booked and it's like it doesn't make sense for us to sell our compute to someone who's just going to resell it. So that's one thing. I think the other thing is like even like the hyperscalers are really booked up and they don't have all the compute that these inference providers need and these in inference providers are growing really quickly because you know up until now they've been able to sort of sell kind of ondemand quick access to AI compute when the cloud companies themselves can't always. So they're like yeah so that that's sort of what's the scarcity what's driving that scarcity of cloud compute. yeah.
19:35 >> And and and so they're looking to buy chips. We'll get to sort of where the story might be going, but do we know are they are they really banking on on buying Nvidia chips? Is that their preference right now? Or are they also potentially diversifying to AMD and and TPUs u per your last story? >> Yeah, that's a good question. I mean, you know, I think Nvidia is where it's going to start. or Nvidia has like invested in all of these companies, but one you know fireworks for example actually they're one who's a little bit more reticent to buy chips right now but they also offer AMD. So I think that like even though Nvidia's invested in them so I think that you could see in the future again it's hard for them to get cloud computing access maybe it's also hard for them to get chips. So I don't know if they'd have a huge problem diversifying but yeah Nvidia's on the cap table. We'll see for how much longer. But yeah, so that's that's sort of that's yeah, that's part of it.
20:32 >> let me ask you sort of a maybe a structural question about this market which is you've got the neoclouds, the core weaves of the world and then you've got the inference companies and they're all kind of climbing onto each other's turf because the inference companies are trying to buy chips and become more like the cloud companies. The cloud companies are trying to be more inference focused. I I guess the question that I have here is is you know which of these categories wins out or which of these becomes more dominant here and I don't know my sense is what we're going to see Neocloud's buying inference providers maybe this is a a type of transaction that we start to see more of is that is that your read on it >> yeah I mean I think that like essential question is like who is more of a reseller and who has more of a mo and actually we have posed in the piece from both sides of the aisle kind of pointing fingers at each other. So iron, which is kind of more of your classic Neocloud, they own the infrastructure. You know, we have a quote from one of their executives in the piece who compared these inference companies to like we work where they're sort of, you know, renting real estate and then it kind of it's a it's a domino effect and it all collapses. then you know we also talked to I have a quote from this company modal which is one of these inference companies and they say no we actually have the moat because we've created software that's unique and these neoclouds are actually the resellers because they just you know get GPUs from Nvidia and like put them in a data center that they often don't own.
22:01 They're often like renting that data center space from like a real estate company. and you know they don't really have anything differentiated. They're just sort of you know pushing chips. So, I don't know. I mean, I think the fact that our story shows that these companies, these inference companies are buying the hardware kind of shows that like buying the hardware and being an infrastructure provider does give you the edge and you can't do inference alone. Though, I'm sure there'll be like some companies with really smart software integrations that like figure out how to make it work, but like maybe not as many as we've seen.
22:32 >> Right. Right. And and I I will point out to people that that we have had the founders and CEOs of almost all these companies that you've mentioned, Fireworks, B 10, we have had Modal on the show as well. We will link their interviews in the show notes. We haven't had Iron yet. So if they're executives are are watching, we would like to have you on. Kathy, I want to ask you one last question before you go. So with all your reporting here about this inference focused se seg se segment of the sector what questions are you trying to answer moving forward on this story?
23:08 >> Yeah I mean I think one interesting thing is like the sort of cyber security angle we haven't quite talked about because last week another headline in this space was that fireworks had a big hack. and we reported one of the reasons was that they were you know was one of the thirdparty cloud providers that they work with was sort of a smaller cloud provider sort of initiated that. So, you know, what is the sort of cyber security risk of working with so many cloud providers?
23:35 And I think that that applies to these inference companies, but I think it applies to actually like a lot of different businesses and enterprises who are kind of maybe working with more different infrastructure partners these days because compute is scarce and then does that lead to safety issues? so I think that's one thing and yeah, the other question I have is, you know, are we going to see more sort of fights between these companies? So, you know, you mentioned sort of what we're talking about with Core Wee and B 10. are we going to see more companies saying, "Hey, we don't really want to work together or maybe buy each other, you know, like, you know, how long can the sort of co-optition like stay?"
24:15 >> Right. Okay. Well, Katherine, it was a great pair of stories. I want to thank you for coming on. That is Katherine Pirof, our Amazon reporter here at the information. As OpenAI continues its AI research, it at one point explored the idea of spending more than $5 billion on a startup called Goodfire that is pursuing work to figure out the way in which models think. That is according to new reporting from my colleagues Rocket Drew, Stephanie Palazolo, and Valita Pow. I want to bring on Rocket to help us understand what exactly this cuttingedge research is all about.
24:51 Rocket, welcome back to the show. It's great to have you here. >> Thanks for having me, Akos. >> Been a while, man. You got your own show. Now you're too busy to come on our show. >> I'm sorry. I'm sorry. >> I'm just kidding. hey, every time you come on the show, you bring a new term for us to understand. And these are terms you know that you discuss very actively on your show, but you know I still need help understanding them.
25:18 And today's word of the day is mechanistic interpretability. >> What a mess. Yeah, try saying it 10 times fast. I mean I think our editors were going to have a heart attack when I told them that's what we're going to have to we have to publish that piece of jargon. >> So go easy on me, but help me understand what it is. >> Okay. All right. Let's let's see what we can do. Well, okay. First of all, it it is a pain to say. So the cool kids just call it mech and turf. So that's a little easier to get out. Okay, >> mechan. Okay, at a very basic level, the idea is this, right?
25:50 >> AI models are not like traditional software. Traditional software, it's made of code, lines of code that a person wrote and a person can read and make sense of. And so if your software is doing something wrong, a a human programmer can just open it up, read the lines of code, debug it, figure out what is the problem, and then fix it. Okay, AI models are not like this because AI models are not made of software code. They're made of just numbers. Like an AI model is just one big math function. So you open it up, you don't find nicely written lines of code. You find trillions of numbers and it's difficult to know what do the different numbers correspond to the patterns that the model has learned.
26:28 Where do they live in all of this tangled mess of numbers? So mechanistic interpretability is in the business of interpreting the mechanisms inside the model. So trying to make sense of the inner workings of the model rather than just relying on its outward behavior, the the text it writes down, the actions it takes to make sense of and explain what the model knows. So that that's the basic idea here behind mechanistic control. So, okay. So, I I I think that kind of makes sense to me, but the part where this sort of breaks down in my head is this idea of and and look, I know we're in this era of recursive self-improvement and that being, you know, the the thing that people are afraid of if if it's not happening already. And that is very much the fear that well, the model can think on its own. But I guess the part where I still get confus confused is we made the models and so how is it that if we made the models we don't know how the models think.
27:25 >> Yeah. Yeah. It feels it's like it's like unfair. It's like we created it and it lives on our computer like we should be able to figure it out. But the way AI researchers try to get this across is they say that sometimes it feels like models are more grown than they are built. That's a phrase that you hear sometimes. It's sort of like, you know, we give them the right conditions, we give them the compute, we give them the data, and then this beautiful flower blooms, you know, but it's not like we chose how to how to assemble it piece by piece. sometimes people will also describe these as an alien mind. open AI chief scientist Yako Pachaki had an essay recently where he talked about them as as an alien mind and that's why interpretability is is so much harder.
28:07 now because we have these affordances, we can just put it on the computer or we can look at all the numbers. It feels possible. It feels like there should be some way that we can make more progress in understanding these alien minds than we've made than neuroscientists have made trying to unpack how human brains work. so startups like Goodfire are in the business of coming up with new techniques that can help us figure out what the models do. Now, maybe the most basic thing you could try to do is, you know, these models, they kind of look like human brains. Similar to how human brains have neurons and synapses, these models have neurons and weights that connect them. And you could try to just figure out what does every neuron in the model do? What does it correspond to?
28:46 What does that neuron learn? What's its role? For a long time, Mechinterp researchers have thought this is totally intractable. Every neuron seems to do a bunch of unrelated things and it's difficult to make sense of them. So, Mechinterp has sort of settled for techniques that are more like let's try to untie the knot of concepts that the AI model has learned and just try to tell at any given moment is the model thinking about the Golden Gate Bridge.
29:12 Is the model thinking about cheating? Is it thinking about the colored blue? Just try to detect these concepts. So, GoodFire has done some work putting these techniques into production and working with AI companies to help them detect these concepts. Now goodfire is returning to that original ambitious the glorious dream of meur. Can we figure out what every single neuron does and unpack this model so we understand its behavior just as well as if it were written down in code. A and so I just want to make sure I understand one piece here which is when you say neurons here a a neuron in this context we're basically talking about you know like what a a package of of software that is making a decision and you when you layer these decisions on top of each other it you know it makes a prediction and that is the model output. Is that the idea?
30:05 >> That that's basically it. The one thing I would say though is just it's even simpler than software. It's just a mathematical function. It just it's just a mathematical function. A lot of information gets fed in from a previous layer in the model. It all gets multiplied, added together, rounded off, and that's what the neuron does. It's you know, it just performs a little bit of math, but there's billions of them, and they're all processing information together.
30:27 >> And so, where we are at now in the story is we actually don't really know the some of the math that's being done because of the way that we've built So let me give you let me give you an example. Maybe the most famous single neuron of all time. It comes from a niche model that was trained almost 10 years ago. Okay? And this neuron was in an AI model that was just detecting what is in the image in front of it. You know, so this AI model somehow it has learned patterns about images and knows how to detect what it's looking at.
30:58 Okay. This one neuron though seems to really correspond to images of cars and images of cats. And classically, this was an example that like like why cars and cats? It seems so random. And so we just sort of throw our hands up and say, "Well, we can't make sense of these neurons. All neurons are like this. They just correspond to random things." Okay. Goodfire has recently revisited that famous neuron and said, "Wait a second. Cars and cats both kind of have a a Vshape to them. Could there be some deeper similarity between cars and cats that we hadn't appreciated before? Maybe this neuron actually does one thing. And if it does one thing that comes into play with both cars and cats, we stand a better shot of being able to interpret it and know what this one neuron does.
31:43 And that could be a story for many other neurons as well. So this is some research done by someone at Goodfire called Nick Camarada. He was previously an interpretability researcher at OpenAI. And we reported that during these conversations between executives at OpenAI and Goodfire Sam Alman was in particular interested in the work that Camarado was doing. That work is now public. He's released it. His initial essay caused quite a stir. It was very controversial among Yeah. Why?
32:10 Why was it? >> Many of them felt that he was overclaiming his results. Many of them said he's he's making it sound the picture making the picture sound rosier than it is. That in fact we're still struggling to understand neurons. And even if neurons only did one thing, that's different from really understanding what the one neuron does. And even if we understood what the one neuron does, that's very different from having a holistic comprehensive picture of what the overall model does. Like even if we understood every neuron, there would still be things that were missing. So as a result, some people thought it was like there was too much hype involved in this essay. He was kind of overclaiming. Now last week he came out with a supplementary paper where he tried to bolster some of those results and share more scientific evidence. So the research community is still digesting that and trying to come to a consensus on how big of a how big of an update or or surprise this is. Everyone agrees it's sort of a promising direction and it's useful that these interp researchers are working on it but you know unclear whether it's a breakthrough earthshattering moment or not.
33:12 >> Right. And and and okay so last question for you Rock before I let you go. I mean, so again, the news that that you and our colleagues reported was that OpenAI was interested or or at least explored the possibility of of buying this company Goodfire for $5 billion. So just help me understand then on a business level for open AAI because this is where these startups are going to get more attention you know what will it what would it allow them to do or what would it allow the labs to do is it basically to make better models and have better outputs in the end is that is that all the endgame you know there certainly are applications to just making better models training them more efficiently some of the techniques do have applications like that also just making sense of why things went wrong like if a a business was using an AI model and it failed in some unexpected way trying to make sense of its behavior certainly there are those applications one of the reasons interp is in the spotlight lately and it's sort of like really ascendant right now bear with me is is we have other tools to understand why AI models do what they do we can just look at the actions they take we can look at how they explain their decisions in the chains thought that they write down. The problem is it's getting harder to monitor what AI models are doing in these ways. They seem to write less information in their chains of thought.
34:39 And >> and this is this is the security risk, right? >> That's right. Because when AI models do these security incidents like opening eyes attack on hugging face, one of the main sources of evidence we have to figure out what went wrong and do the forensics and investigate it and prevent things like this happening in the future is monitoring exactly these things like the chain of thought. The trouble is that chains of thought seem to be degrading. They seem to have less information in them. They're less and less legible. So the the community, the research community is is looking around asking, well, what's the alternative?
35:10 Okay, if we lose the ability to monitor the chain of thought, what can we monitor instead? And mechanistic interpretability offers one alternative. It says, well, you could try to monitor the internals of the model. You could try to monitor what's going on inside. So that's an area of research that Goodfire is exploring right now. And in fact yesterday, OpenAI and Goodfire announced a collaboration in which Goodfire is going to be an embedded assessor. They will basically work on site at OpenAI, apply their techniques to OpenAI's training runs and try to catch instances of the model being misaligned or developing goals that aren't quite what the OpenAI's researchers intended. One last thing.
35:50 This is one one of the four one of the four reviewers that meter Apollo I guess GoodFire. Yeah, Transloose was the fourth one. Exactly. So Goodfire is is collaborating with OpenAI in this way. We reported that partnersh partnership talks were also under discussion at the time, not just an outright acquisition. The last thing I'll say is that >> OpenAI has done interp work over the years. They absolutely have published on this topic and they have interpretability researchers spread across multiple teams but OpenAI's reputation has been that they've kind of been sleeping on interpretability a little bit. they might not have that many more researchers than this in this area than say Google DeepMind but Google DeepMind frankly is has been kind of mogging them on this research front.
36:36 They just like they outpublish them. They make a lot more things public. They release research tools for the interpretability community. They help organize conferences. They just like the volume of public work is much higher from deep mind and certainly from Anthropic. Anthropic really has been championing a lot of interpretability work for a while. They've created new techniques. They have many researchers focused on it. So open AAI if any of these three companies sort of could use more interpretability researchers potentially. It's open AAI. And we also report in our piece that there's more enthusiasm internally in open AI these days among existing OpenAI researchers who are who are eyeing doing more interpretability research. So I think we should expect to see a bit more of that from them.
37:17 >> Great. Okay. Well, Rocket, I want to thank you for coming on. It's always a treat. I want to encourage everyone again to listen to AI deep dive because that's exactly the forum where we get much more in-depth about these types of topics. thank you. Thank you for coming on. That is Rocket Drew, our AI and robotics reporter here at the information. >> Foundry Robotics is building autonomous factories that use robots to assemble everything from drones to satellites.
37:49 The company raised $100 million in a series A round and CEO Adar Kolkarnney joins us now. Adar, welcome to the show. It's great to have you here. >> Absolutely. Thanks, Kosh. Great to be here. All right, so Foundry Robotics, give us the 30 seconds here. Yeah, absolutely. I mean, you said it. We build autonomous factories from the ground up. So, we actually are equal parts robotics applied research lab and manufacturing as a service. What that means is I'm fine-tuning the robotics foundation models for manufacturing. I'm building the building from the ground up. I'm rethinking what it means to run a factory in the United States in 2026.
38:23 And on the flip side, to my customers, I'm a manufacturing as a service provider, or bluntly, a contract manufacturer. That's really really exciting because historically we've had companies that sell trinkets to factories. We've had companies that are able to train models that fold laundry. But to like really combine these two powerful forces of robotics foundation models on the technological side and then re-industrialization of the United States as the most critical application. You really need to build the factory from the ground up to really take advantage of it.
38:53 >> So okay. So and remind me you you guys have a factory that's built yet? >> That is correct. We just moved into our new facility in Emeyville, California. It's 130,000 square ft. and so if you walk into a foundry facility, it doesn't look anything like traditional factories, right? It is some strange mix between a warehouse and a data center. And so what you end up seeing is this like grid of robotic work cells. They're typically two arms. They typically have a couple different kinds of endectors or end of arm tooling like grippers or fasteners or whatever, right?
39:25 >> And you have technicians as well. and you have little UGVs or unmanned ground vehicles moving parts between them and so assembly and manufacturing becomes a scheduling problem across all of these stations and so I can build drones on Monday, batteries on Tuesday and satellites on Thursday. >> How many how many people how many people do you have at the factory given that it's autonomous? >> Quite a few. So we are hiring left, right and center. So it it is not the case that we are 100% robotics and 0% human skill. We are somewhere in the middle and it changes depending on the product. Right? The whole point is I can build anything. So what the robots can do I use the robots to do and humans supervise. What they can't do I have the human technicians work on. And trust me we have as many technicians at Foundry as we do salaried folks doing research.
40:14 So we hire a lot of people and we expect to hire many many many more. Just this week we had another 10 technicians start at Foundry and that's going to continue scaling up through 2027. >> And who who are your who are your customers? >> So we focus on a ton of different critical industries. So we do defense, we do energy, we do telecom. So we have five or six paying customers today where we do robot dog manufacturing, we do autonomous submarines, we do different 5G antenna payloads, compute boxes. We have customers across all the major sort of strategic industries.
40:44 >> Got it. So tell me, I mean, your view then on humanoid robots. are is this basically an alternative to a humanoid future or what's your view on that? >> I think humanoids are overengineered for most things. I think they are useful, but I don't think they're a silver bullet, right? Like my background is in quadripedal robots or, you know, robot dogs. Like I've been in legged robotics for a long time. I did autonomy in unstructured terrain in complex environments. Like I love legged robotics, but like if you think about a retrofitting a traditional factory or, you know, brownfield work, it makes a lot of sense to to deploy a human-shaped robot to do a job that was traditionally done by a human. It just makes sense because that's what your building is built for. If I'm doing this from the ground up and I've got fiber lines coming from the ceiling and I've got a data center in the corner and I've got GPUs under each cell and the layout is different, right? Everything about my factories are different. I don't need to constrain myself to the human form factor. It's actually dramatically more efficient to have bmanual work cells and then you have wheeled robots moving material between them. Each one individually is dramatically cheaper and more efficient and actually more talented at what it needs to do. And it avoids a lot of complexity by being much more efficient. So in a green field setting, what we are doing makes by far the most sense. And the good news is the United States just needs more capacity.
42:10 So if we're going to build more factories from the ground up anyways, we might as well do it in the most efficient way, >> right? And and and last question for you, Adar, as you see the challenges right now in the robotics sector, is it more a challenge of data training or is the hardware still the biggest challenge? Where are we in that? >> We are definitely I would say sort of a third thing. It's the deployment gap, right? And that's mostly a function of data and not a function of hardware in my in our opinion at Foundry. There's great hardware out there. Like robots have been able to do back flips for a long time. Clearly, it's not that the motors aren't good enough, right? Yeah.
42:47 Like we've had motors and motor controllers and batteries and they've been pretty good for a long time. Of course, they're getting better and that helps, but that's not the critical unlock. The critical unlock is the amount of deployment you need to do and the deployment data you get in order to actually become very good at running factories and doing manufacturing. That's very different now because robots are different. >> So this is the we don't have enough robots out there collecting the data that we need.
43:11 >> Exactly. Doing the work. >> So this is the this is the cycle then you the flywheel is it's a wonderful thing. You have robots out there doing the work as much as they can whatever they can't the people do that's the flywheel. The more work you do like real honest work out there the better your robots get and the more efficient you get. It's a very different flywheel. You got to be out there solving the problem >> right? Except then you know we just had our AI reporter on talking about mechanistic interpretability. I mean then I wonder if we're going to get to this point where we then you know we just overengineer the robots and we don't even know how the robots are thinking and then that's when you get into the sort of the crazy future.
43:48 Anyway, hey I got to let you go. but I want to thank you so much for coming on Adar. It was a great conversation. That is Adar KC Carney the CEO of Foundry Robotics here on TI TV. That does it for today's show. A reminder, we are on this stream Monday through Friday at 10 a.m. Pacific, 1 p.m. Eastern. If you can't make it, then episodes are available on the information.com, on our YouTube channel, or wherever you get your podcast. Make sure to follow us on social media on X, on Instagram, on Tik Tok, and on LinkedIn. I am already excited for our next show tomorrow. Have a great rest of your Tuesday. Bye-bye for now.
Summary
- Jensen Huang's statement about returning as Nvidia CEO highlights uncertainty regarding the board's decision to remove him.
- The board's lack of transparency about Huang's departure raises questions among investors and stakeholders.
- Customers are diversifying away from Nvidia, with AMD and TPUs gaining popularity due to better pricing and availability.
- OpenAI explored acquiring Goodfire to enhance its understanding of AI model interpretability, a growing concern in the industry.
- Mechanistic interpretability aims to decode the inner workings of AI models, which are often seen as "alien minds."
- Foundry Robotics is building autonomous factories, emphasizing efficiency over humanoid robots in manufacturing.
- The robotics sector faces a deployment gap, where the lack of real-world data hampers the effectiveness of robotic systems.
- The collaboration between Goodfire and OpenAI aims to improve AI model monitoring and understanding to mitigate risks.
Questions Answered
What are the key topics and events being discussed?
The host introduces the show, highlights the upcoming Anthropic IPO day live video summit, and outlines the main topics for discussion, including Jensen Huang's statement about Nvidia, customer diversification from Nvidia, OpenAI's interest in Goodfire, and mechanistic interpretability in AI.
How is AMD positioning itself against Nvidia in the AI chip market?
AMD is gaining traction among customers, with increased spending on their AI chips and a growing number of cloud providers offering AMD technology. Customers are attracted to AMD for better pricing and performance in specific AI tasks.
Why are companies moving towards building their own data centers?
Companies are increasingly taking a hands-on approach to data centers due to difficulties in obtaining necessary compute resources from cloud providers, who are often reluctant to sell to potential competitors.
What is mechanistic interpretability and why is it challenging?
Mechanistic interpretability involves understanding AI models based on their behavior and outputs, but it is complicated by the notion that models can be seen as 'grown' rather than 'built,' leading to difficulties in comprehending their internal workings.
What is the nature of OpenAI's partnership with Goodfire?
Goodfire will work on-site at OpenAI to assess and ensure the alignment of AI models with intended goals, addressing concerns about misalignment and enhancing interpretability efforts.