transcribe

Why OpenAI Could Spend $5 Billion to Understand AI

The Information · 13m · transcribed 3d ago
More from The Information Business
𝕏 Share ▶ YouTube 📥 PDF 🤖 .md

Section Insights

# 0:00

Introduction to Mechanistic Interpretability

What is mechanistic interpretability and why is it important?

Mechanistic interpretability refers to understanding how AI models function, which is challenging because AI models operate differently from traditional software. While traditional software is built from readable code, AI models are complex mathematical functions that can behave in unexpected ways.

  • AI models are not like traditional software; they are complex mathematical functions.
  • Understanding AI models is crucial for debugging and improving their performance.
  • The term 'mechanistic interpretability' is often shortened to 'mech and terp' for convenience.
# 2:39

The Challenge of Understanding AI Models

Why is it difficult to understand how AI models think?

Despite being created by humans, AI models can seem to think independently, leading to confusion about their inner workings. Researchers often describe these models as 'alien minds,' making interpretability a significant challenge.

  • AI models can appear to think independently, complicating our understanding.
  • The analogy of AI models as 'alien minds' highlights the difficulty in interpretability.
  • Understanding AI models is compared to the challenges faced by neuroscientists studying the human brain.
# 5:18

Understanding Neurons in AI Models

How do neurons in AI models function?

Neurons in AI models perform mathematical functions by processing inputs from previous layers, but the complexity of these interactions makes it difficult to fully understand their behavior. Each neuron contributes to the model's output through a series of mathematical operations.

  • Neurons in AI models are essentially mathematical functions that process information.
  • Understanding individual neurons does not provide a complete picture of the model's behavior.
  • The complexity of interactions among billions of neurons complicates interpretability.
# 7:57

Research and Developments in Interpretability

What recent developments have occurred in AI interpretability research?

Recent research has revisited the understanding of specific neurons, suggesting deeper connections in their functions. However, even with insights into individual neurons, a comprehensive understanding of the entire model remains elusive. The research community is still evaluating the significance of these findings.

  • Recent research has provided new insights into the functions of specific neurons.
  • Understanding individual neurons does not equate to a holistic understanding of the model.
  • The research community is cautiously optimistic about new developments in interpretability.
# 10:36

OpenAI's Collaboration with Goodfire

What is the significance of OpenAI's collaboration with Goodfire?

OpenAI's collaboration with Goodfire aims to enhance model interpretability by monitoring the internal workings of AI models. This partnership allows Goodfire to apply its techniques directly to OpenAI's training processes, helping to identify and correct misalignments in model behavior.

  • OpenAI is collaborating with Goodfire to improve AI model interpretability.
  • The partnership focuses on monitoring internal model processes to catch misalignments.
  • Goodfire's techniques may lead to better model performance and understanding.

Transcript

0:01 As OpenAI continues its AI research, it at one point explored the idea of spending more than $5 billion on a startup called Goodfire that is pursuing work to figure out the way in which models think. That is according to new reporting from my colleagues Rocket True, Stephanie Palazzolo, and Valita Pow. I want to bring on Rocket to help us understand what exactly this cutting edge research is all about. Rocket, welcome back to the show. It's great to have you here.

0:26 >> Thanks for having me, Akash. >> Been a while, man. You got your own show. Now you're too busy to come on our show. >> I'm sorry. I'm sorry. >> I'm just kidding. hey, every time you come on the show, you bring a new term for us to understand. And these are terms you know that you discuss very actively on your show, but you know, I still need help understanding them. And today's word of the day is mechanistic interpretability.

0:55 >> What a mess. Yeah, try saying it 10 times fast. I mean, I think our editors were going to have a heart attack when I told them that's what we're going to have to we have to publish that piece of jargon. >> So go easy on me, but help me understand what it is. >> Okay. All right. Let's let's see what we can do. Well, okay. First of all, it it is a pain to say. So the cool kids just call it mech and terp. So that's a little easier to get out.

1:18 >> Okay. >> Mech and terp. Okay. At a very basic level, the idea is this, right? AI models are not like traditional software. Traditional software is made of code, lines of code that a person wrote and a person can read and make sense of. And so if your software is doing something wrong, a human programmer can just open it up, read the lines of code, debug it, figure out what is the problem, and then fix it. Okay. AI models are not like this because AI models are not made of software code. They're made of just numbers. Like an AI model is just one big math function. So you open it up, you don't find nicely written lines of code, you find trillions of numbers. And it's difficult to know what are the different numbers correspond to, the patterns that the model learned. Where do they live in all of this tangled mess of numbers? So mechanistic interpretability is in the business of interpreting the mechanisms inside the model. So trying to make sense of the inner workings of the model rather than just relying on its outward behavior, the the text it writes down, the actions it takes to make sense of and explain what the model knows. So that that's the basic idea here behind mechanistic interpretability.

2:25 >> okay, so I I I think that kind of makes sense to me, but the part where this sort of breaks down in my head is this idea of and and look, I know we're in this era of recursive self-improvement and that being, you know, the the thing that people are afraid of if if it's not happening already. And that is very much the fear that well, the model can think on its own. But I guess the part where I still can confuse is we made the models and so how is it that if we made the models we don't know how the models think?

2:58 >> Yeah, yeah, it feels it's like it 's like unfair. It's like we created it and it lives on our computer. Like we should be able to figure it out. But the way AI researchers try to get this across is they say that sometimes it feels like models are more grown than they are built. That's a phrase that you hear sometimes. It's sort of like, you know, we give them the right conditions, we give them the compute, we give them the data, and then this beautiful flower blooms, you know, but it's not like we chose how to how to assemble it piece by piece. Sometimes people will also describe these as an alien mind.

3:31 OpenAI's chief scientist Ilya Sutskever had an essay recently where he talked about them as as an alien mind. And that's why interpretability is is so much harder. Now, because we have these affordances, we can just put it on the computer, we can look at all the numbers. It feels possible. It feels like there should be some way that we can make more progress on understanding these alien minds than we've made than neuroscientists have made trying to unpack how human brains work. So startups like Goodfire are in the business of coming up with new techniques that can help us figure out what the models do. Now, maybe the most basic thing you could try to do is, know, these models, they kind of look like human brains. Similar to how human brains have neurons and synapses, these models have neurons and weights that connect them. And you could try to just figure out what does every neuron in the model do? What does it correspond to?

4:19 What does that neuron learn? What's its role? For a long time, Mechanistic Interpretability researchers have thought this is totally intractable. Every neuron seems to do a bunch of unrelated things and it's difficult to make sense of them. So, Mechanistic Interpret has sort of settled for techniques that are more like let's try to untie the knot of concepts that the AI model has learned and just try to tell at any given moment is the model thinking about the Golden Gate Bridge? Is the model thinking about cheating? Is it thinking about the color blue? Just try to detect these concepts.

4:51 So, Goodfire has done some work putting these techniques into production and working with AI companies to help them detect these concepts. Now, Goodfire is returning to that original ambitious the glorious dream of Mechanistic Interpret. Can we figure out what every single neuron does and unpack this model so we understand its behavior just as well as if it were written down in code. >> And And And so I just want to make sure I understand one piece here, which is when you say neurons here, a neuron in this context, we're basically talking about you know, like what a a package of of software that is making a decision and when you layer these decisions on top of each other, it you know, it makes a prediction and that is the model output.

5:37 Is that the idea? >> That that's basically it. The one thing I would say though is just it's even simpler than software. It's just a mathematical function. It just It's just a mathematical function. A lot of information gets fed in from a previous layer in the model. It all gets multiplied, added together, rounded off, and that's what the neuron does. It's you know, it just performs a little bit of math, but there's billions of them and they're all processing information together.

6:01 >> And so where we are at now in the story is we actually don't really know the some of the math that's being done because of the way that we've built these models. >> me give you let me give you an example. Maybe the most famous single neuron of all time. It comes from a niche model that was trained almost 10 years ago, okay? And this neuron was in an AI model that was just detecting what is in the image in front of it. You know, so this AI model somehow it has learned patterns about images and knows how to detect what it's looking at. Okay, this one neuron though seems to really correspond to images of cars and images of cats.

6:38 And classically this was an example that like like why cars and cats? It seems so random. And so we just sort of throw our hands up and say, well, we can't make sense of these neurons. All neurons are like this. They just correspond to random things. Okay, Good Fire has recently revisited that famous neuron and said, wait a second, cars and cats both kind of have a a V shape to them. Could there be some deeper similarity between cars and cats that we hadn't appreciated before? Maybe this neuron actually does one thing. And if it does one thing that comes into play with both cars and cats, we stand a better shot of being able to interpret it and know what this one neuron does. And that could be the story for many other neurons as well. So this is some research done by someone at Good Fire called Nick Camarata. He was previously an interpretability researcher at OpenAI.

7:26 And we reported that during these conversations between executives at OpenAI and Good Fire, Sam Altman was in particular interested in the work that Camarata was doing. That work is now public. He's released it. His initial essay caused quite a stir. It was very controversial among AI >> that? Yeah, why why was it controversial? >> of them felt that he was overclaiming his results. Many of them said he's he's making it sound the picture making the picture sound rosier than it is. That in fact we are still struggling to understand neurons. And even if neurons only did one thing, that's different from really understanding what the one neuron does. And even if we understood what the one neuron does, that's very different from having a holistic comprehensive picture of what the overall model does. Like, even if we understood every neuron, there would still be things that were missing. So, as a result, some people thought it was like there was too much hype involved in this essay. He was kind of over climbing. Now, last week he came out with a supplementary paper where he tried to bolster some of those results and share more scientific evidence. So, the research community is still digesting that and trying to come to a consensus on how big of a how big of an update or or surprise this is.

8:35 Everyone agrees it's sort of a promising direction and it's useful that these Interp researchers are working on it, but you know unclear whether it's a breakthrough earth-shattering moment or not. >> Right. And and and okay, so last question for you, Rock, before I let you go. I mean, so again the the news that that you and our colleagues reported was that OpenAI was interested or at at least explored the possibility of of buying this company Good Fire for $5 billion. So, just help me understand then on a business level for OpenAI because this is where these startups are going to get more attention, you know, what will it what would it allowed them to do or what would it allow the labs to do? Is it basically to make better models and have better outputs in the end? Is that Is that all the end game?

9:26 >> You know, there certainly are applications to just making better models, training them more efficiently. Some of the techniques do have applications like that. Also just making sense of why things went wrong. Like, if a a business was using an AI model and it failed in some unexpected way, trying to make sense of its behavior, certainly there are those applications. One of the reasons Interp is in the spotlight lately and it's sort of like really ascendant right now.

9:52 bear with me. Is Is we have other tools to understand why AI models do what they do. We can just look at the actions they take. We can look at how they explain their decisions in the chains of thought that they write down. The problem is it's getting harder to monitor what AI models are doing in these ways. They seem to write information in their chains of thought at any >> this is the this is the security risk, right?

10:15 >> That's right. Because when AI models do these security incidents like OpenAI's attack on Hugging Face, one of the main sources of evidence we have to figure out what went wrong and do the forensics and investigate it and prevent things like this happening in the future is monitoring exactly these things like the chain of thought. The trouble is that chains of thought seem to be degrading. They seem to have less information in them. They're less and less legible. So the the community, the research community is is looking around asking, well, what's the alternative? Okay, if we lose the ability to monitor the chain of thought, what can we monitor instead?

10:49 And mechanistic interpretability offers one alternative. It says, well, you could try to monitor the internals of the model. You could try to monitor what's going on inside. So that's an area of research that Goodfire is exploring right now. And in fact, yesterday OpenAI and Goodfire announced a collaboration in which Goodfire is going to be an embedded assessor. They will basically work on site at OpenAI, apply their techniques to OpenAI's training runs, and try to catch instances of the model being misaligned or developing goals that aren't quite what the OpenAI's researchers intended.

11:23 One last thing >> This is one one of the four one of the four reviewers that Meter, Apollo, I guess Goodfire and yeah, Transluce was the fourth one. Exactly. So Goodfire is collaborating with OpenAI in this way. We reported that partnership partnership talks were also under discussion at the time, not just an outright acquisition. The last thing I'll say is that OpenAI has done inter work over the years. They absolutely have published on this topic and they have interpretability researchers spread across multiple teams.

11:55 But OpenAI's reputation has been that they've kind of been sleeping on interpretability a little bit. They might not have that many more researchers than this in this area than say Google DeepMind. But Google DeepMind frankly is has been kind of mogging them on this research front. They just like they out publish them. They put make a lot more things public. They release research tools for the interpretability community. They help organize conferences. They just like the volume of public work is much higher from DeepMind and certainly from Anthropic. Anthropic really has been championing a lot of interpretability work for a while. They've created new techniques. They have many researchers focused on it. So OpenAI, if any of these three companies sort of could use more interpretability researchers potentially, it's OpenAI. And we also report in our piece that there's more enthusiasm internally in OpenAI these days among existing OpenAI researchers who are who are eyeing doing more interpretability research. So I think we should expect to see a bit more of that from them.

12:50 >> Great. Okay. Well, Rocket, I want to thank you for coming on. It's always a treat. I want to encourage everyone again to listen to AI Deep Dive because that's exactly the forum where we get much more in depth about these types of topics. Rocket, thank you for coming on. That is Rocket Drew, our AI and Robotics reporter here at The Information.

Summary

OpenAI considered acquiring a startup called Goodfire for over $5 billion to enhance its understanding of AI model interpretability, specifically through a concept known as mechanistic interpretability. This approach aims to decipher the inner workings of AI models, which are fundamentally different from traditional software, making it challenging to understand how they function.

- Mechanistic interpretability (or "mech and terp") seeks to explain AI models' internal mechanisms rather than just their outputs.
- AI models consist of complex mathematical functions represented by vast numbers, unlike traditional software with readable code.
- Researchers liken AI models to "alien minds," as they can behave in unexpected ways that are difficult to interpret.
- Goodfire is developing techniques to identify and understand the roles of individual neurons in AI models, which could lead to better interpretability.
- A controversial study from Goodfire suggested that neurons might correspond to specific concepts, sparking debate about the validity of such claims.
- OpenAI's interest in Goodfire reflects a broader need for improved interpretability to enhance model reliability and safety.
- Collaborations between OpenAI and Goodfire aim to monitor AI model behavior more effectively, especially in light of security concerns.
- OpenAI has historically lagged behind competitors like Google DeepMind in interpretability research, prompting a renewed focus on this area within the organization.

Questions Answered

What is mechanistic interpretability and why is it important?

Mechanistic interpretability refers to understanding how AI models function, which is challenging because AI models operate differently from traditional software. While traditional software is built from readable code, AI models are complex mathematical functions that can behave in unexpected ways.

Why is it difficult to understand how AI models think?

Despite being created by humans, AI models can seem to think independently, leading to confusion about their inner workings. Researchers often describe these models as 'alien minds,' making interpretability a significant challenge.

How do neurons in AI models function?

Neurons in AI models perform mathematical functions by processing inputs from previous layers, but the complexity of these interactions makes it difficult to fully understand their behavior. Each neuron contributes to the model's output through a series of mathematical operations.

What recent developments have occurred in AI interpretability research?

Recent research has revisited the understanding of specific neurons, suggesting deeper connections in their functions. However, even with insights into individual neurons, a comprehensive understanding of the entire model remains elusive. The research community is still evaluating the significance of these findings.

What is the significance of OpenAI's collaboration with Goodfire?

OpenAI's collaboration with Goodfire aims to enhance model interpretability by monitoring the internal workings of AI models. This partnership allows Goodfire to apply its techniques directly to OpenAI's training processes, helping to identify and correct misalignments in model behavior.

© transcribe · For agents Built with care and craft by Gokul Rajaram