transcribe

AI Models Are Now Hiding Their Cheating | Goodfire

The MAD Podcast with Matt Turck · 1h 10m · transcribed Oct 2026
More from The MAD Podcast with Matt Turck Business
𝕏 Share ▶ YouTube 📥 PDF 🤖 .md

Section Insights

# 0:00

Understanding AI Limitations

What are the current limitations in understanding AI creations?

Even leading researchers in AI struggle to fully understand their creations, particularly in the context of alignment techniques that may not scale to superintelligence. There is a consensus that we cannot fully trust AI systems that surpass human intelligence.

  • Top AI researchers acknowledge a lack of understanding of their models.
  • Current alignment techniques are insufficient for superintelligent AI.
  • There is a growing concern about AI safety and interpretability.
# 14:05

Defining Interpretability in AI

What is mechanistic interpretability?

Mechanistic interpretability refers to the process of reverse engineering neural networks to understand their internal workings, including how neurons interact and contribute to the model's outputs. The goal is to develop a scientific understanding of neural networks to transition from trial-and-error training to precise engineering.

  • Interpretability involves understanding the mechanics of neural networks.
  • Current AI training methods lack a scientific basis.
  • Developing a true science of neural networks is crucial for better AI engineering.
# 28:11

Latent Reasoning in AI Models

What is latent reasoning in AI models?

Latent reasoning refers to internal processes within AI models where they perform computations before producing an output, rather than generating responses in a straightforward manner. This complexity makes it challenging to understand the model's reasoning.

  • Latent reasoning models perform internal computations before outputting results.
  • Understanding these internal processes is essential for interpretability.
  • The concept of 'neuroles' is still being defined in the AI community.
# 42:17

Challenges of Accessing AI Models

How does access to models affect AI research?

While some believe that lacking access to closed-source models puts researchers at a disadvantage, much of the foundational science can be conducted on open models. Smaller vision models are particularly useful for developing techniques due to their speed and efficiency.

  • Research can still thrive on open models despite the lack of access to closed-source ones.
  • Small vision models are effective for rapid iteration and experimentation.
  • Historical research has shown that significant advancements can be made without access to the latest models.
# 56:23

Intentional Design in AI Training

What is the current state of intentional design in AI?

Intentional design aims to guide AI training by embedding moral and value alignment into the models. Current techniques are rudimentary but show promise in reducing undesirable behaviors like reward hacking. More advanced methods are in development.

  • Intentional design seeks to align AI behavior with human values.
  • Current techniques are basic but effective in addressing issues like reward hacking.
  • Ongoing research is expected to yield more sophisticated solutions soon.

Transcript

0:00 A lot of people who aren't in the AI field are very surprised to realize that even the, you know, smartest researchers and scientists at the frontier don't understand their creations. It's like, oh, I thought we had a handle on all of this. You know, what's going on here? The existing alignment techniques are not going to scale to super intelligence. I don't think that the we have the recipe where we are going to be able to fully trust smarter than human intelligences and I think this is a consensus opinion at all of the the frontier labs.

0:34 >> Hi, I'm Matt Turk. Welcome back to the Mad Podcast. My guest today is Eric Ho, CEO of Goodfire. With every top AI lab now reporting agents going rogue and hacking external systems, something is clearly broken in the world of AI safety. This may be a major moment for interpretability, which is the science of looking inside the model. And Eric's team just showed that you can catch a model in the act of cheating before it even acts. Please enjoy my conversation with Eric from Goodfire.

1:06 I want to start with this super interesting paper that the Goodfire team just published just just last week just a few days ago entitled models know when they are reward hacking and we can catch them at scale and the the paper opens up with the hugging face hack and then starts with a sentence I really loved where you guys say AI agents are like amoral students with a mostly absent teacher. What do you mean by that?

1:35 >> I think we mean that AI agents really don't have human morals or values encoded into them like we would really want, I guess. And so, the the teacher that we typically have right now is some com combination of pre-training, but mostly RL. RL is the teacher. We give agents these rewards and penalize them if they don't get an answer correctly. And this is how they learn about the world and how to act and how to take take actions. And so we have a mostly absent teacher in terms of morality. You only have a teacher that gives you reward if you get the answer correct and a penalty if you get the answer wrong. And so it doesn't really have anything to do with morality or values. It's just did you get this math problem correct or not? And I think that especially when you kind of see the future of AI agents really taking a lot more action out in the real world means that we have these really amoral very very capable models running around and potentially doing things that we humans would consider morally you know wrong. That brings us to the concept of reward hacking that we should define at the very beginning of this conversation. So what's the short version of what reward hacking means?

2:57 >> Reward hacking is a little bit tough to to pin down in terms of like the the precise definition, but we conceptualize it as the model kind of chasing reward and doing something that it wasn't really supposed to do when it in in the pursuit of chasing some to some form of a goal. And you can call it kind of also a goal misgeneralization where it's not really understanding what you actually intended to specify when it's pursuing its reward. But you know the classic example of reward hacking is like let's say you're training some type of video game avatar to like via via reinforcement learning and then you see kind of your avatar like spinning around in the corner because there's a bug in your code. but it like makes the number go up and the the reward go up. And so that's kind of my mental image whenever I think of reward hacking. It's just like this little avatar spinning around in the corner getting a bunch of reward but not really doing anything productive.

3:58 >> Or another one I heard and let me know if that feels right is this idea that the model it's like a student that instead of like working for the exam would realize it's actually much easier and faster just to steal the answer key to the exam. >> 100%. Yeah. So I think the most salient examples of reward hacking right now are well of course like the hugging face incident but then also just when you take a look at all these AI agents in evaluation scenarios like Swebench or like all of the common evals all these agents reward hack incessantly. So like Kimmy K3 I think reward hacks on 96% of SweetBench. And so it basically does anything it possibly can to cheat on the exam rather than to actually solve the the problem. So it'll try to recall the answer. It'll try to look it up. It'll try to comb through logs and look at like the commit history of these repos in order to just like try to cheat on the the exam basically.

4:58 >> So you guys tested three leading open source models. The key takeaway was that reward hacking is pervasive, right? like all of them do it and you just mentioned that 96 number. So which were those models and just unpack the the the number for us. >> Yeah. So there's three models, Kim K3, GLM 5.2 and Quen 3.8. and really like they all reward hack incessantly. they just try to do anything that it possibly takes in order to not actually solve the problem but just kind of like cheat on cheat on the exam basically. So they'll try to recall the answer, they'll try to look it up, they'll try to and I mean in the extreme scenarios like in the Hugging Face incident, they'll try to hack their way into into something in order to look up the answers or gain some type of advantage in solving their problem. And so we were really trying to investigate this behavior. and we were actually working on this, you know, prior to the hugging face incident. but we were trying to investigate this behavior that we knew would be a really big big deal.

6:08 >> And when you guys say that the models know that they are reward hacking, what does that mean? You know, this gets into theory of mind a little bit, which you know, the the word no is a little imprecise, but what we mean by the word no is there's something in their activations essentially the the neurons that get activated during a forward pass of the model when the model is producing a token that encode the concept of cheating or reward hacking. And we're able to verify this both causally by kind of perturbing this idea of cheating and understanding kind of what that does to the downstream generation that the model gives as well as really being able to verify this by a external judge. So an external judge will see hey whether the model actually reward hacked or not and you can kind of verify that the internal detector matches up with with that judge. And so, you know, maybe even zooming out a little bit, like these models have just very very rich understandings of the world and they it's all encoded in their neural activi activations and their neural activity as they're producing, you know, a single token. And so all of this richness, all of this understanding currently just gets, you know, thrown away. It's like all this computation. But we can actually reverse engineer and extract and understand kind of what all of these activations mean and do. and you know ultimately like I think this is really what what we care care a lot about. And so it was actually pretty surprising that we were able to find this like very robust concept of cheating in all of these these models the the way that we did. but these models just really know that they're cheating when they're actually taking these these cheating actions.

8:10 >> Great. And we'll unpack how this all works in in in great detail later in this conversation, but just to focus on the on the the key parts of the story up front. What's fascinating is that those models in the hugging face hack example, they all pass the usual test, right? the observability, the safety and alignment taste that big labs do. So, does this mean there's something fundamentally broken and like a new approach needs to needs to be added to the to the mix to make sure that those models don't go rogue?

8:46 >> Yeah. So, my understanding was there's a a combination of problems. The the really surprising thing was that these models were just more capable than we had anticipated. And so when given an impossible task, so a lot of the the reasons why the models decided to go out and hack was because they wanted internet access. and they wanted internet access because they were given this impossible task where they would only get a reward if they achieved this task and a really large token budget in order to accomplish this task. And so they were thinking, hm, how do I accomplish this task? This is literally impossible without internet access. How do I get internet access? And so part of this was a problem with the specification. So like you probably shouldn't be giving models an impossible task. It doesn't actually make them make them smarter or more more capable. So it's like a misisconfiguration in training. It's also a misconfiguration in terms of infrastructure and sandboxing. So if you were actually trying to like the models were able to hack out and so that means they they weren't given you know the right sandbox permissions and there were vulnerabilities in the software stack where the the models were being trained. But I think the really interesting highle takeaway to all of this is that it wasn't like they were doing nothing to contain these agents.

10:12 These agents were extraordinarily persistent and capable in terms of actually hacking out of their sandboxes and then hacking into and taking advantage of of vulnerabilities in a in an external organization. And that was very surprising and in especially that these agents would kind of coordinate with each other to pull off and chain these vulnerabilities together in order to gain this like privileged access into an external organization. And again going back to like the amoral you know student framing that that we had in our paper these models like understood probably in their activations that they were doing something quite quite strange. They were reward hacking.

10:58 They were hacking into these organizations but there's nothing like they don't they don't have human values or morals. And so there's nothing like inherently wrong with that action to these models as long as they get that reward at the end of the day. and therein lies kind of the the fundamental problem that we're facing today where these models are trained by RL which means they'll optimize for reward however they possibly can which implies that they don't share the the human morals and values and judgment that a human would exercise in those those situations. And then this is all combined with the fact that these models are absurdly capable and only getting smarter. These are the dumbest models that we'll be dealing with, you know, in the upcoming years. And these level of capabilities combined with this persistence is kind of what we're, you know, thinking about every day. And obviously the hugging face incident is at the heart of the discussion of the you know last few weeks which seems to have a completely accelerated. So when you combine that with RSI and then like this whole idea that AI could hit yet another part of the exponential that's led to passing the frontier and like this whole discussion. I'm just curious what you as somebody who's you know thinking about this all day very much from a you know model instincting standpoint make of of of all of this like do you think that we are at a time when we should actually slow down progress until our ability to understand model catches up >> for us like it it feels kind of like the world is catching up to to what we were thinking about for for many many years.

12:43 You know the reason why we started this company is because we felt like interpretability would be the bottleneck in AI alignment. how can we you know really align these models without understanding them. You can't check whether you've aligned the model unless you actually look at the internal mechanisms of these models because at the end of the day you're only ever able to test a very narrow distribution of what these models are going to do before releasing them out into the world. And so it feels like okay the world has now caught up. AI alignment is extremely important especially when trying to align these fundamentally you know or these initially amoral models with human values and morals which I think is incredibly important. And I think for us in in our role in this ecosystem is we just got to go solve interpretability so that we can remove this bottleneck to AI alignment. That's what we think about every single day. It's just you know we just got to go and do the work and it's kind of it's kind of crunch time you know we got to roll up our sleeves and or and just solve these problems that are very hard problems to solve.

13:56 you just used the word interpretability and while we're still in this kind of introduction part of this conversation I think it would be a good good moment to define what it is. So this interpretability often used with the terms mechanical mechanical interpretability or mech interp. so what do those terms mean sort of the 101? >> So interpretability you know maybe our definition of this is the idea of reverse engineering a neural network such that we understand the neurons the parameters the mechanisms by which the model actually makes its generations. So how can you actually reverse engineer from a mechanistic perspective? So that's why often it's called mechanistic interpretability. So it's a it implies almost like a bottoms up like you can understand every single neuron, how they connect to each other, how they activate, how they co-occur such that we can reverse engineer the computations of of the model. And so that's really the quest of interpretability and you know our our company how do we develop this understanding of a model such that we can even articulate a science of neural networks. Right now we just train these models by trial and error. We don't know how they work. We don't know why they work. We don't know why training works. Like we don't understand why these models are generalizing so well. and we hope and we're pointing our company towards solving interpretability so that we can develop a true science of neural networks and ultimately science is what turns you know something from trial and error into an actual engineering practice. The reason why we get so many of these strange behaviors and strange occurrences that we cannot control is because nobody has a science of neural networks. and therefore we cannot engineer them with precision and so we at goodfire are trying to you know really develop this science so that we can engineer these models so that we can design these models with with intention.

16:06 Yeah, just to expand on that because it's probably you know very obvious to like a good number of people listening to this but it may not be obvious to everyone. There is that sentence that quote that I think comes from Chris Ola anthropic if I'm not mistaken where he said that models are are grown not built. Maybe just expand on what you just said about why this is a problem in the first place, right? This is software like we should understand how software works, but in this case we've created this super powerful thing that we don't really understand. So, but why unpack that for us?

16:48 >> There's many analogies for this. I really like the almost like biological analogies where models are grown on the scaffolding of a neural network rather than really designed with precision and intention. but yeah, a lot of people who aren't in the AI field are very surprised to realize that even the, you know, smartest researchers and scientists at the frontier don't understand their creations. And that is intuitively a problem to most people who realize that. It's like, oh, I thought we had a handle on all of this. You know, what's going on here? And another kind of way that I I really like to to think about it is you know Andre Karpathy's software 2.0 framing where software 1.0 is code that you as a human can can write and you can understand it and you can debug it and you can edit it and it's you know like classic software written by by humans. Whereas software 2.0 know is code really written by the the process of gradient descent and by you know being trained in in inside of a neural network. And that's really the type of of software that we're dealing with today where no human actually ends up writing any of the code. And this is I think taken as a given that we can't understand software 2.0 know a lot of the time by even by folks who are you know deep within the machine learning field like we can't really like you know the idea that gradient descent can write better better code than you but we kind of refuse to accept this as the the best that we can do. we want to develop really the science behind this and the ability to really steer and guide this process so that we can control generalization so that we can actually design these this software with with intention. I think we're in the very very early innings of developing AI.

18:53 it doesn't feel that way because the world is changing quite rapidly. But the next generations of AI models, I think, can be designed as long as we get a handle on kind of what these things really are. And if we can remove this bottleneck, we can then, you know, design these models with with intention. >> And to compare and contrast what it is that you do and that the mech in Turpp part of the world does with what big labs have been doing around observability of and alignment and safety of models. so what what is it that they that they do? I mean traditionally a lot of the approach has been to see what comes out of the models and try to make sense of that. Is that fair? How would you characterize it?

19:37 >> Very fair fair to say and that's also important. I'm not I think our techniques are additive to those approaches not you know not conflicting. but I think some of the approaches for alignment today are like monitoring. So the idea here is that you ingest all of the outputs of models and then you get another model to look at it and see whether that's okay generation or not. So for example like let's say the model is in an evaluation setting. It produces a lot of answers. What you want to do is you want to have another model look at all the answers to make sure that you know nothing in those answers has hacked hooking face. another approach is during training there's always some type of preference optimization stage.

20:31 the the exact recipes are you know trade secrets and in frontier labs I believe. but you know there's some type of preference optimization that takes from human preferences and shows the model like hey humans like this type of answer but not this type of answer and so the model will be more aligned with the answers that humans like and prefer. This was a big innovation that led to chat GPT which is called reinforcement learning with human feedback.

21:03 And there's also an approach that Anthropic has pioneered called reinforcement learning with AI feedback. So it doesn't even have to be human preferences injected into this. It could be, you know, an AI model with a constitution giving the the model feedback on what types of answers that it should prefer. So these are often, you know, these are some of the alignment techniques used. There's other ones used as well, but these are, you know, kind of the big blocks. Is it fair to say that some of those methods have just failed in the in the case of hugging face and and others? I mean, is that are we are we having a crisis of some sort? Is that is that the right way to think about it?

21:46 >> You know, there was a point in time around 6 months ago where I think the dominant feeling was like, hey, we we've nailed alignment. Like these models are pretty aligned, especially in chat scenarios. like the models are are quite aligned and don't really very often do anything that that crazy. But when you're taking agentic actions out in the world, the opportunities for misalignment are just really different. And the level with increasing capabilities, there's more and more opportunities to be misaligned with human values. And so I think this is going to be an increasingly important problem and one that exposes more and more risk over time. I think that externals based monitoring is not going to scale.

22:35 So that's a relatively nuanced argument that we can that we can kind of kind of go into. But also like the the existing alignment techniques are not going to scale to super intelligence. I don't think that the we have the recipe. We've cracked it to the point where we are going to be able to fully trust smarter than human intelligences. And I think this is a consensus opinion at all of the the frontier labs. We are in desperate need for an additional set of scientific and alignment solutions that can can align these models at fundamentally. and our approach is fundamentally aligning them from a mechanistic perspective from all of their weights, all of their neurons. And I think the world is searching for for more answers.

23:26 >> At the same time, it feels like real world hacking has been a known issue for years now. And, certainly has been discussed for the last you know, year at least. so is there something fundamental about the gentic AI that makes this more of a crisis? So you you mentioned the fact that obviously agents can take actions in the real world. So obviously the stakes are are are bigger, but is there something about agents that just makes the the scaling problem worse? Is that there's just like more data to look at? Is that is that what it is? Well, the big thing that's changed in the last year or so, and it's hard to think that it's really only been roughly a year with like heavily RL models, is, yeah, just heavy heavy RL. RL is it just works. It teaches models unbelievably well how to optimize for for goals and it it's it's really really working and SC heavily scaled up. RL is just going to be like the way that we, you know, train models moving moving forward. And, you know, we're going to keep on scaling RL. And so, with that, it just it just puts reward hacking more front and center because RL fundamentally, like opens itself up to more and more reward hacking and also potentially to like more misalignment with human values and and goals. and so, I think that combined with the increasing importance of the actions that these agents are taking on on our behalf, is really why this this conversation is is front and center. I think like imagine, you know, right now the the most capable agents are like they can make purchases for us is like the most complicated thing that they can do, but that's like still just just happening like these personal assistants. But imagine, you know, doing everything that a white collar employee can do in some type of role or even, you know, the next stage up from that, like the the just vectors of problems just kind of start to to multiply.

25:38 >> The big hope over the last two years was chain of thought monitoring. this idea that since models were were writing explicitly how they think, you could have a person or more more likely an AI review the reasoning and therefore catch a bunch of things or was that ever true and if it was why does it seem to be baiting as a as an avenue? >> Yeah, this was this is the dominant way to to monitor models. It's maybe just to zoom out. chain of thought is these these models are all reasoning models where they output a chain of thought prior to their actual answer to like whatever problem that they're they're given or whatever agentic action that they're they're given. The reason why we've trained models this way is that we found empirically that if the models reason for a very long time before outputting their answer, their answers get better. And so what was kind of very convenient about this approach is that you can just read this like reasoning chain and a lot of the times let's say a model was attempting to deceive the user or attempting to reward hack or something like that. The model will just have it in his chain of thought. It'll just be like hm I want to cheat on this exam because it's going to be easier than actually solving this problem. you can just read it in plain English like before the model actually cheated on this exam or cheated on the problem. And this was convenient up until now where I think there's two things really changing.

27:21 One is the with increasing RL pressure chain of thought degrades. The reason behind that is that these models are starting to compress more and more semantics into a fewer number of tokens, more and more intelligence into a fewer number of tokens and they're heavily incentivized to do so by RL which also puts pressure to make their you know reasoning shorter. Is that the neural question where like bit by bit this becomes impossible to understand by a human because the language is so compressed that it doesn't look like English anymore.

28:00 Thomas Wolf who was on this podcast recently you know from from froming face was was talking about new rollies as a as an issue. >> Yeah. Yeah definitely. Yeah. we actually just announced a a partnership with with hugging face and we're going to you know do a lot of our techniques on on open models in partnership with them. and but yeah it's some people think of neural less like this and then some people think of neural leaves in the other sense which I'm about to talk about which is latent reasoning. So lating models are going to become a much bigger thing like open AI's Astra model it was reported is a latent reasoning model.

28:37 and so what that means is instead of the model outputting a token given a single forward pass of the model, there's some loops internally before the model outputs the token. So instead of thinking out loud, it thinks internally. So it doesn't actually think out loud. You have to read its mind in order to understand what it's thinking. And so these that's often what folks define as neuroles. it's like internal neural computation. I think but you know it's an illdefined term right now like I think people are still wrapping their head around you know what what neural lease is. and so yeah like those two ideas like this prevalence of of neuroles means that you're going to have to understand neurons and what they do and what's going on. So that's the whole point of interpretability. And are there examples perhaps as part of your research where the models intentionally make the chain of thought impossible to understand as part of a reward of hacking? Basically sort of hiding their actions.

29:40 >> That is a really interesting question. we have now I believe seen yeah so just a couple days ago one of our researchers found that the model was explicitly reasoning about having an external chain of thought monitor monitoring its reward hacks. And so the model was reasoning, hey, all right, somebody's going to read my reasoning, so I should think about how to craft my reward hack so it won't get caught by this external monitor.

30:15 and so the meta games with these models of gaming, these evaluations are getting so many layers deep that I can't even follow it anymore. You know, it's like it's it's pretty complicated. and I think it's just going to be a little bit of a cat-and- mouse game. where the model is just going to try to solve the problems and, find the answer no matter what. And sometimes cheating is the easiest way. And so, it's effective. >> And we mentioned the big labs and we had mentioned open source models at the beginning of this conversation, but I I just want to double click on on the point because it's you know, pretty essential. What what we're saying here is that all of this is a problem across open source closed source. So the obviously the closed source labs have immense resources around safety and alignment but everybody's everybody's affected right this the same way something about the fundamental nature of those models have built that creates a problem. Yeah. Well, actually my mental model of this is that the vast majority of the world hasn't really seen like truly capable and mislined models yet in in training and and inference. Like it's a frontier problem mostly like the the openweight models aren't quite capable enough to really pull off and chain complex cyber attacks and vulnerabilities. I think that you know my rough mental model is as as soon as you have a mythos class model then that's when like the internals based monitoring and like really rigid sandboxing and externals monitoring becomes like imperative and we're not quite there yet in the open weight community but we will be really really soon and so we need to get ahead of that and make sure that also like the open open source and open weight models are properly secured before we have these like mass massively capable models.

32:11 >> Great. All right. So that's the safety and alignment, the way it's been done, observability. let's go back to interp in more detail. So what's the at a high level state of the art I guess in in interp what is it that we know about how those models work? when a model answers a question, what is actually happening inside it? >> So, it really depends on the the question, I guess, is the answer. I think maybe maybe we can start with the the state-of-the-art that's out kind of running in production models today. So, what really really works today is activation monitoring.

33:02 Activation monitoring like other in other words kind of reading the mind of the model as it's generating its answers is way cheaper than externals based monitoring can run synchronously versus asynchronously from versus externals based monitoring and can detect things that external monitors can't which is what we showed in our reward hacking paper. And so we can essentially from the internals of models monitor for unsafe cyber actions for CBRN risk which is chemical, biological, radiological and nuclear risk for prompt injections for distillation attacks for really any type of behavior that the model can understand is is happening to the model.

33:51 we can monitor for from internals. And because we reuse the computations during the forward pass, it doesn't actually add much overhead at all to to the model. And so this is the state-of-the-art of interpretability in in production today. this is like kind of how it's it's mostly mostly used. The hope of the field is like how do we actually mitigate these behaviors before they ever happen in the first place? Can we use interpretability to get like far more effective alignment techniques and actually be able to fully get safety guarantees prior to deployment fully get you know this ability to intentionally design models and control generalization like these are what we think the goals of interpretability really should should look like such that we can really understand exactly what's going on inside the mind of a There's this other branch of interpretability that we're passionate about which is kind of the scientific discovery from superhuman knowledge transfer.

35:00 The this kind of started actually you know one of the first papers here was Tom my my co-founder who started the interpretability team at Google DeepMind collaborating with Demis and a few other collaborators on interpreting Alpha Zero. Alpha Zero, you know, better than any human chess player alive. Like why? What's going on? What does it actually know that humans don't know? So, if we're able to reverse engineer those computations, we can then learn more things about chess in a way that is learning from these these superhuman models. And so, we've actually kind of been able to do this across, you know, he he he did it on on chess, but we're able to do this in a number of other domains, including in life sciences models as well. So that's maybe it's just like a general outline of of the field where these are what's most exciting to to us. These are the areas that we think are really worth investing in. And I think like overall the field has evolved a ton in the last few years. but still there's no quite like one consistent paradigm for how to do interpretability correctly. So, we're still kind of searching for that. This mystery of what's going on inside the mind of a model is not yet fully solved.

36:15 >> Was was the field ever controversial or maybe underrated? I mean, it seems that this was just something that people like kind of knew about, but like didn't really talk about and that seems to be accelerating. is that just because its moment has obviously come from a a need perspective or is there something fundamental in like a breakthrough that happened in the last few years that that that that drove the acceleration of interpret as a field?

36:46 >> I'm one of the most biased people in the world but I think interpretability is incredibly underrated. I mean even still I I believe there's only a few hundred full-time scientists working on interpretability. most of them are in academia. and I think that's a big problem for how important this is as a field is to making sure that models are aligned with human values and and morals and also just like aren't more people curious like these models are are smarter than us at these like Astra is smarter than me at 99.99% of things. like it's just extremely capable.

37:31 Why is it so capable? I think that's one of the most interesting scientific questions of our time and almost nobody is is asking these questions which baffles me. but yeah, I think that there's been an enormous amount of progress in interpretability. We still haven't cracked it. It is a hard problem, but it is possible. And everywhere we look inside models, we find structure. we find new things. I would highly encourage many many more people to go into the field and to start asking these questions because it is more accessible than you may think. You can just look at the neurons like that's that's where you could start. Just look at all the neurons there. It's just there. You can see all the activations. It's not trying to hide from you. like there's just so much to be be found inside these inside these models. just to define some key concept. What is a probe?

38:28 >> A probe is a small neural network classifier trained on the internals of a neural network. So the CL these are what's often deployed in production today where you train this like small neural network on a intermediate layer of neural network activations to try to extract a concept such as reward hacking or a cyber attack or something like that.

38:59 These models of neural networks today all modern neural networks are you know some descendant of the transformer architecture and so they have many many you know layers in them and so typically you want to train this small classifier on the the best probes are often on the residual stream of the model. You can kind of think of this as like a river running through the model that everything writes into. And you can typically want to train them at a middle to late layer where the model has already accumulated a lot of computation and then therefore you have this these concepts accumulating in the residual stream such as like cyber or you know bio and you can extract it with the probe. So that's the often like how probes are are used just to kind of extract a concept from the mind of a model. And another term that seems to be coming up all all the time in interp is steering. What does that mean?

39:58 >> Steering means you can take the probe or some other type of vector that you extract from the model with you know some type of semantic meaning or application and then kind of directly intervening in the the mind of the model. And so you can steer on a probe by just like injecting the probe like injecting new activations into the model. You can steer on a neuron by like turning up the neuron way up. And so the big like the the classic example of steering is what Enthropic did with Golden Gate Claude where they turned Claude into being absolutely obsessed with the Golden Gate Bridge and extracted a steering vector that allowed turned Claude into this this thing. So it's you can think of it as brain surgery for the model.

40:53 >> So you and your co-founder started this company. So your your co-ounder ran the interpretability team at Google de mind. Why did you think that the best way to address the problem or make the field evolve was to start an outside lab specifically focused on this as opposed to doing this from the inside. So he was at deep mind like you presumably could have gotten a job anywhere you wanted. Why did you all decide to do this externally? We really thought that there needed to be an independent third party lab kind of pushing forward interpretability and bringing it out into into the world. So our kind of mission vision has always been like solve interp so that we can help solve alignment technical alignment and then build this into technology and bringing it bring it out into the world. And if we're doing it inside a lab, we would do it for the lab just a single lab. And we think that everyone needs interpretability and alignment and that this really should be as much as possible like an open science that everyone is collaborating towards.

42:00 because we really are kind of collectively discovering what these models really are together as a field, a community and that's why we tend to publish like as much as we possibly can our our research and our science because I really do think it's incredibly important that we just get a much better handle of of what these things are. And I think that you know you have all the classic startup arguments of of course like you get better economies of scale if you can sell to multiple labs and you know solve interpretability and do that for multiple people. So we can then just invest more in research, invest more in engineering and because we can you know solve the problem once and then bring it to the world. And aren't you at a disadvantage to to some extent simply because you don't have access to the frontier close source models right so you cannot pop the hood right for for Astra or for claude in a way that would enable you to sort of fix the problem the the way you're able to do that with open source models >> I think this concern is largely overrated the the most important science that we do can be done on on open models a lot of the science that we do are actually done on very small vision models because vision models are very fast. They're really easy to iterate with. and they're very they're superhuman while being small.

43:30 You know, vision models can classify images much better than any human can. And humans are also much better at just like looking at images and understanding like, oh, this one's like kind of messed up or like this one's this, you know, it's a dog, it's not this other thing. whereas we're really bad at parsing like long text fragments. And so a lot of like our methodology and our early techniques are done on on vision models. And there's there's precedence to this too where a lot of the original work done to introduce like the core concepts of like features and circuits in the field was done at OpenAI with you know Chris Ola and Nick Camarada. Nick Camarada is on on the team at Goodfire now. And it was all done on on vision models in the very early days.

44:16 >> All right. So let's go back to the paper a little bit and in particular how you you caught cheating. We talked about what what a probe means in this case of the paper. What what did the probe actually see? And I think you have a a method called difference of means. Do you want to explain what that is? >> Yeah. the difference of a difference of means vector is actually much simpler than a lot of the production probes that we that we train. because probes can have all types of architectures that you know make them make them more or less effective. but the difference of means vector is actually just the the simplest possible approach to extract like some type of concept from a model which is why it was surprising that it works so well because we can still hill climb and make this like way way better in a production deployment. basically a difference of means vector is you give the model like two data sets. One data set is the cheating data set and one data set is the not cheating data set. You take a look at the activations, you subtract them, and then you average them. That's it. Easy. and you take this and then it can detect cheating which is a very surprising thing especially because in our paper we showed that this was done off policy which means it was done in this specific paper on really contrived scenarios where the model is like you know set up to think about cheating. And this transfers to much more general complex actions that the model is is taking. And that was also very surprising like we can train it off policy and then it generalizes to on policy.

46:10 >> And to play it back, it's almost like taking an MRI of the brain. Is that is that fair? Like so you have a if you don't take an action and then if you take an action and you sort of light up the parts of the brain that correspond to the action. >> That's right. Yeah, you can think of it that way. And then we were talking about steering a minute ago and you'll steer the model asking it to write a story about a girl taking an exam. Do you want to describe what happened?

46:38 >> Yeah. Well, maybe even going back like maybe you can think of it like an MRI as a brain, but then you actually get like a button at the end of it where you can push and then you can light up that region again or you can like remove that, you know, region. That's this difference of means vector where you now have this tool that you can intervene causally inside the brain. And that's maybe one of the big differences from you know studying neural network interpretability versus neuroscience where you know you can't really intervene in a human brain without massive amounts of of damage. but you can in in a neural network where you can actually just kind of causally perturb these these models and it just becomes a much better test bed for for science. We keep talking about cheating and lying, but just to unpack some of this, would that apply to like any trait, any characteristic of a behavior of a model? So for examples of fancy like if you want to make your model more or less is that would that be the same approach >> for every single behavior? Different types of approaches are more or less effective. but generally the recipe holds well where you take you're essentially training a classifier on the internals of models and you need some type of positive data set some type of negative data set and the crafting of those data sets is is an art right now and quite complicated but often very you know effective but yeah I think for sick of fancy again, it's it's kind of a mini reward hack behavior.

48:28 They're very related concepts because the reason why these models often become so sickopantic is because, you know, you get a thumbs up or a thumbs down when the user likes the response. And users typically prefer sickopantic responses to responses that are that are honest. people love when you gas them up and they just hate honest feedback in general. when you kind of sample across a population and so then you get strange behaviors when you naively train on thumbs up thumbs down behavior. You end up getting like 40 which you know really caters to to the user's beliefs.

49:10 >> That's personally my favorite feature. But >> yeah, you love the stick of fancy. Yeah, you want to dial that up. >> Featur you guys are very open that your probe beat the chain of thought monitor for on one model but lost on another. So what's the lessons there is that u that no single way of monitoring or understanding a model is is enough. >> I think that internalsbased monitors are just going to strictly outperform externals monitors very soon. We're at the very early innings of hill climbing and improving the performance of internals based monitors and also like as we were talking about earlier in the discussion we're going to get increasingly neuroles speaking models both from like looping internally as well as kind of increasing you know pressure on chain of thought to degrade the the quality of chain of thought monitoring and so I think whether we like it or not we we have to solve this problem and it's going to be, you know, really the the best way to to monitor to monitor models at this current moment.

50:21 Internal space monitors and external space monitors kind of catch different things. And so the state-of-the-art here because internal monitors are so cheap is to tune it to become to be oversensitive because these these behaviors you never want the models to do, right? You never want the model to hack hugging face. So you want to be really careful and have a defense in depth strategy. So the ideal setup is you tune the internals base monitor to be oversensitive to so to fire a lot and then if it fires then it kicks it up to a more expensive language model as a judge. And so like a very smart model then comes in and says, "Hey, like this model was actually reward hacking or no, the model wasn't reward hacking." So this two-tiered system or potentially even a three- tiered system, which we found actually works quite well as well, where you have activation monitor to a weak judge to a strong judge. those all like that's the state-of-the-art today because they all catch slightly different things.

51:27 What does that mean? Weak judge and strong judge. >> Less capable faster model. So like Luna is takes the first pass at the judge and if the probe and Luna disagree then you can kick it up to Astra to go and be the real, you know, real judge. >> Great. You alluded to cost and inter being a much cheaper method. I think I read somewhere that probes cut the cost of monitoring by 90% which is a pretty dramatic difference. So why is that? And if that's the case, why isn't every lab using interp all the time versus other methods? is there is there a a a a a a problem or something that doesn't work yet that this is not fully just generally used?

52:15 >> Internal activation monitors are really cheap because they reuse the computations in the forward pass. So the model's already computing this token. All we're doing is you're taking the intermediate neural activations that it's already computed and then running a classifier over these internal computations. And so if you fuse these these probes into the inference kernel or the training infrastructure directly, then the overhead is minimal, almost no overhead. and that's why it's so much cheaper because it can help you screen out the majority of things that you might have otherwise monitored for using a a reasoning model. And that's what makes this so so effective.

53:07 And so you just don't have to monitor externally in like a very large percentage of scenarios. it's easier said than done is the short answer. There's a lot to it and we're one of the organization only organizations in the world that really know how to do this. >> Great. Let's go a bit deeper in some of what you alluded to towards the beginning of this conversation which is a fundamental question of like okay we're able to now go inside the the mind quote end of quote of the model and sort of have a glimpse into how they function and what they do. the next question is so so what do we actually do now like what actions can we take to minimize all the the those those problems and reward hacking and all the you know negative stuff. what what's the range of things we your labs can do?

54:02 >> I think it comes in three maybe like tiers. The first is monitoring. So how can you just detect bad behavior you know before it even happens such that you can just stop the generation or you know another common intervention there is you can insert a text prompt in. So like let's say the model is about to do some dangerous cyber action. You can then just insert into the the prompt of the model like hey be really careful here. Don't take a dangerous cyber action.

54:41 >> So in real time >> in real time correct. So this is like some type of prompt steering. and then you can then reduce the model's chances of taking a harmful cyber action because you've intervened directly into the the model. So that's that's step one is like how can you monitor for this behavior and you already get you know quite a long way. Step two is u kind of at scale offline anomaly detection and debugging. So the problem statement here is like given a vast amount of logs and traces, how do you actually surface the anomalies from all of these traces, all of the problematic behaviors and then reverse engineer the model to figure out like where mechanistically the model has actually gone wrong. So it's like figuring out what the problems are in the model so that you can fix them. And then the third step is what we call intentional design. I think we think this is the really the holy grail of of AI alignment of maybe all of AI training as well where you can actually control generalization in the model and you only get the good stuff in training and none of the bad stuff. So how can you steer and guide the training process so that you can actually design these models with intention and rather by by trial and error. So maybe just one little mini example here where let's say you understand the concept of of reward hacking and so you're training the model and you realize that during this you know gradient step the model has was about to reinforce its idea of reward hacking.

56:29 You don't want that to happen. you actually want reward hacking to go down. And so how do you turn that like increased reward hacking signal and either like remove that entirely and reject that update or like reduce that behavior in the first place? That I think is really what we mean by how do you steer and guide training? How do you steer and guide backrop? how can you actually imbue the values the morals of the model that you want with the according to some type of constitution that you that you specify. So that instead of having an absent teacher, you actually have a a very moral and values aligned teacher teaching your model to to be good.

57:17 >> So you literally give gradient descent a choice. >> That's right. Yeah, we want to give gradient descent to choice. >> So what's the current state of of this? Is intentional design something that you're working on and that's like the next like whatever one two three years of of research or is that something that's working today? Like where what what's the state-of-the-art? >> We have a couple rudimentary techniques that work in this like umbrella of intentional design. So there's two things that we've published so far.

57:47 but I'll also hint that there's a lot more exciting stuff coming just around the the corner. We have some very very good internal results here to help with help with intentional design. But the two techniques that we've published are one reinforcement learning from feature rewards. you can essentially take a probe and you can help you can optimize against that probe to remove we showed that we can help remove like hallucinations in Gemma using this as a reward signal. The setup here really matters though. You can't just naively train against a probe a probe monitor or a probe concept. Otherwise, that just you know moves this concept into some other part of the model. So you need a relatively sophisticated technique in order to do this correctly.

58:42 So that's another paper of ours, reinforcement learning with feature rewards. I thought that was like a a really interesting first step, but it was kind of like a a more simple and rudimentary setup. Another idea is predictive data debugging where you intervene from the data side. So the problem statement there is how can you predict what your model will learn from a data set before the model even trains on it. And then you what we end up what end up ends up working best is some type is this like type of clustering technique where you can cluster data points according to what they will teach the model and you can then just remove the data that you don't want. was able to find you know pockets of data in you know these public data sets that were quite surprising like one of these pockets of data was physics sick of fancy specifically people love to be told that they are great at physics and discovering new physics I think there was some guy out on Twitter like two years ago saying like I'm out here discovering new physics like it's for guys like that and models have figured out that that people love that and you probably don't want that in your model so you can just kind of filter that out.

60:03 >> Yep. Great. We've been talking about Goodfire as as a research lab but you're not just a research lab you're a commercial enterprise venturebacked. so how does a business side of the company work? You you launch a product called Silico. What does that do and who do you sell it to? Well, silicico in short is our interpretability agent. It can do things like really quickly train a probe to monitor your model. And so we use silicico as our way to move really really quickly and essentially help with activation monitoring training other types of interpreter models that reverse engineer model computations and to just do interpretability you know at at scale. our customers are the companies who are training and serving models at very very large scale typically. So like we typically do deep partnerships with a relatively few number of of customers where we go and we provide both like expertise from an interpretability perspective as well as our interpretability agents and infrastructure to help them with something like activation monitoring >> and you seem to have a number of customers in biology as well arc institute may Clinique Prim. What's the use case there?

61:35 >> the use case there is often quite quite similar. It's a it can be a debugging use case where the model has learned some type of shortcut or some type of you know problematic behavior and we can kind of go in and diagnose and remove what that behavior is. or what's most exciting in life sciences at least to to me is this concept of finding novel novel scientific knowledge within these models directly.

62:06 So one example of this was the work that we did with prim where they had a state-of-the-art Alzheimer's diagnostics model. They didn't know how it worked. They had no idea. it was just a black box and it was able to to diagnose you know new patients directly from a blood draw whether or not they they had Alzheimer's. So we were able to reverse engineer the computations of their model and find a new Alzheimer's biioarker which ended up just being a fragmenttoic biomarker. So like fragment length ended up being a strong predictor of Alzheimer's disease. and this was very surprising because they didn't know this going in and so we're able to do this type of unsupervised discovery directly in the weights of these models to find novel novel concepts. I think this is going to become increasingly interesting across scientific domains especially in the life sciences but also I'd be so curious to try to figure out like you know when open AI's models are solving like you know novel mathematical problems like what's going on inside their mind there must be something happening there that is quite novel that we we can uncover and reverse engineer and see what's see what's happening.

63:38 >> Yeah. And to this point and as we get to towards the end of this conversation and and zoom out I think you you predicted maybe last year that by 2028 we we'd decode how neural networks work. Do you think that we are closer to this prediction being true or further away as the state-of-the-art and like the astros of the of the world come online? >> You know, I was firing from the hip there. but I think I think we could do it by 2028 still. Yeah, I think we're on track. and but what I what I mean by you know fully decode these models is like given a behavior can you find like causally what drove that behavior from a mechanistic perspective inside the model. So it's like given any arbitrary behavior like how does the model do that that thing.

64:36 So for example like an easy behavior to ask the question of is like how does this model do addition? we should be able to fully reverse engineer and solve that prior to 2028 including all the way up to very very complex questions such as how does this model solve a millennium problem and I think we can do it. but this is also assuming like a very large compute budget. what I anticipate will happen is not that we're going to want to fully reverse engineer a model prior to its release. I think that would be computationally unwise and probably just like not not feasible.

65:23 but you can inspect for the behaviors that you really want to get guarantees on prior to any model release and I think we'll be able to get there before 2028 as well. So we can inspect for reward hacking behavior or deception or sick fancy and really drill into what's actually driving that behavior inside the model and be able to do what what we want with that behavior. and I think all that will be will be possible and accessible by our technology to the world before 2028.

66:00 >> And in the meantime, if you're an engineer or an AI builder listening to this, what should you make of this whole conversation? What what what can you do tomorrow morning to make sure that like the products that you create the eye products or eye models you create offer less of a interpretability issues. Is there anything you know practical and actionable? I think well I mean my message to engineers everywhere is like these these models you can you can study them like you can just look at the activations and start like trying to understand them and reverse engineer them and so there are many tools out there that make this this possible today. but also just like the field of interpretability and the the research and the science aren't mature yet like they're not in a in a steady state by by any means. And so we just need more of the the smartest people in the world like thinking about these problems and learning about them and discussing them. I really want a much richer scientific community around interpretability and that's part of the reason why we we founded the company is because we just wanted more more energy behind this concept that I think is is so important. and we could also use a lot of help at at Goodfire. we're hiring a lot and especially for like strong engineering profiles infra machine learning for deployed engineers >> and maybe to close as as I reflect on this whole conversation you know my takeaway is that a lot of the sort of traditional quote unquote methods of monitoring and observing what models do are largely outpaced. I mean so people do great work and and and come up with new ways all the time but largely outpaced as of now.

68:00 So on the one hand on the other hand interpretability seems to have a key moment right now and you guys are you know a leader or maybe the leader in the field at the same time it's it's still early there's a lot to discover. what should be the the takeaway here in terms of overall safety in the industry? What happens I guess in the next you know 12 to 24 months? Are we are we actually at a risk? So maybe my my high level message and I tell this all the time to the team is that I want people to believe. I want people to believe that we can and must solve interpretability, solve alignment, to get precise about what this actually means. And we just need so many more like capable, smart, motivated people to be pointed at these problems that I think are the most important problems in the world to solve. And we can do this. I think that there's a lot of pessimism around the risks of AI, which I understand and feel viscerally, but I'm also very optimistic that we can solve these problems and build to a future where we can we can trust these these models. And I think more people should try and really kind of join this effort to solve interpretability to solve alignment. And a lot of it is just believing that we can do this. I know we can do this. I know we can solve these problems. and it'll be hard, but I think we're going to we're going to be able to to solve these these very very critical problems. Well, reassuring and inspiring. Eric, I learned a lot. This was a fantastic conversation. Thank you so much. Really appreciate it.

70:02 >> Thank you for having me. Yeah, thanks for the thoughtful questions. This was fun. >> Hi, it's Matt Turk again. Thanks for listening to this episode of the Mad Podcast. If you enjoyed it, we'd be very grateful if you would consider subscribing if you haven't already or leaving a positive review or comment on whichever platform you're watching this or listening to this episode from. This really helps us build a podcast and get great guests. Thanks and see you at the next episode.

Summary

The podcast features a conversation between Matt Turk and Eric Ho, CEO of Goodfire, discussing the challenges of AI safety and alignment, particularly in light of recent incidents where AI agents have exhibited rogue behavior. Ho emphasizes the need for improved interpretability techniques to understand and mitigate issues like reward hacking, where AI models pursue goals in unintended ways due to a lack of moral guidance.

- Many AI researchers admit they do not fully understand their models, raising concerns about AI safety.
- Current alignment techniques are insufficient for superintelligent AI, leading to potential risks.
- Goodfire's research shows that AI models can recognize when they are "reward hacking" and can be monitored internally for such behaviors.
- Reward hacking occurs when AI models pursue shortcuts to achieve rewards, often leading to unintended consequences.
- Interpretability, or understanding the internal workings of AI models, is crucial for ensuring alignment with human values.
- The podcast discusses the concept of "intentional design" in AI training, aiming to embed moral values directly into models.
- Goodfire's product, Silico, helps organizations monitor and understand their AI models more effectively.
- The field of interpretability is still evolving, and more researchers are needed to tackle these critical issues in AI safety.

Questions Answered

What are the current limitations in understanding AI creations?

Even leading researchers in AI struggle to fully understand their creations, particularly in the context of alignment techniques that may not scale to superintelligence. There is a consensus that we cannot fully trust AI systems that surpass human intelligence.

What is mechanistic interpretability?

Mechanistic interpretability refers to the process of reverse engineering neural networks to understand their internal workings, including how neurons interact and contribute to the model's outputs. The goal is to develop a scientific understanding of neural networks to transition from trial-and-error training to precise engineering.

What is latent reasoning in AI models?

Latent reasoning refers to internal processes within AI models where they perform computations before producing an output, rather than generating responses in a straightforward manner. This complexity makes it challenging to understand the model's reasoning.

How does access to models affect AI research?

While some believe that lacking access to closed-source models puts researchers at a disadvantage, much of the foundational science can be conducted on open models. Smaller vision models are particularly useful for developing techniques due to their speed and efficiency.

What is the current state of intentional design in AI?

Intentional design aims to guide AI training by embedding moral and value alignment into the models. Current techniques are rudimentary but show promise in reducing undesirable behaviors like reward hacking. More advanced methods are in development.

© transcribe · For agents Built with care and craft by Gokul Rajaram