Meet the Red Team: How Open-Source AI Is Rewriting Security

0:00 Back with another Bitcoin Policy Institute podcast. we're missing our our third today, Ken Zax Shapiro. this isn't our normal policy hour group, but he's he's with us in spirit. >> So, I was on last week, and I was told that this was the first time you were ever having a guest. So, this is the second time you're having a guest, and I'm back on again. >> So, it's a little muddy here cuz we've got the the Bitcoin Policy Hour, which is like our our weekly show.

0:27 >> And I think this might be like something new for us. You know, we we got a a new a new style here. so, we'll see if if Zax Shapiro gets invited to the next one. >> >> Yeah, that's right. It sounds like you're yeah. >> Or if he appears just as he disappears on the on our other show. >> Right. Right. Right. okay. Well, let's jump into this. We we've got two very special guests with us to dig into this Coldcard issue further.

0:53 we've been keeping a a pulse check on this very closely, staying in touch with some of the the partners or friends of of the institute. and so, we're going to we're going to keep that content coming. We're joined by James O'Beirne and Rob Hamilton today. James, I I'm hearing rumblings that you may be joining the the red team. Do you Do you have anything to share with the audience here? >> I think you got the tense wrong. I don't know if I'm I'm joining the red team. I think I've been loosely affiliated >> Yeah.

1:26 >> Maybe maybe since inception, although who can tell? This is all very shadowy. >> It's all moving very quickly, too. >> moving very quickly. It's all It's all done under a very sleep-deprived state, so. but yeah, I I I I know those guys, you could say. >> I love it. I love to hear it. let let's jump into this. I think for folks who haven't heard yet, there's a a loose term floating around called the red team.

1:53 we're we're sort of you know, aptly naming this episode meet the red team. we don't have the full red team here, but Rob, walk us through sort of what you're working on right now on the Coldcard issue. I think folks have been keeping up with this. We've put out some content over the past week on this, so hopefully folks are familiar, but what what's the current stage of things and and introduce us to the red team here?

2:19 >> yeah, this is my first time at all talking about it. I I actually don't like journalists, but I can count you guys as exceptions. So, this is a good good venue for this. it really hit me and I had a tweet about this. We actually talked about it when I was on last week, where when the initial vulnerability was being kind of sniffed out, I by instinct went to Codex and Claude Code and said, "Look at this." And Codex instantly downgraded me and like gave me the Fable 5 that the from Fable to the Opus 5 model. And GPT was like kind of there. It was like kind of coy. And then Kimmy K3 was just like, "Here's a full printout. Everything is really bad."

3:00 And in that moment I was I had like a a realization of like how powerful this was when Kimmy K3 went open weights last Monday, I started poking around my own internal code base immediately. Our whole team just started immediately going in there and we found a couple like things, but like nothing that was like critical, but like just, you know, software, you know, is complicated and you always want to keep be on top of things. So, we quickly patched our things.

3:21 And that realization of looking at that issue with the Coldcard firmware when the news was breaking made me realize no one had been doing this in their code base anywhere. And I started immediately I did a tweet to the effect of like I'm throwing a couple hundred bucks on my own personal credit card and poking out some like well-used repos. and within the first couple of hours did some responsible disclosures and then confirmations that they were real.

3:47 And it quickly started accelerating. Calle reached out to me saying, "Do you need help?" To which I said, "Yes, I need anything I can do." And he then immediately, jumped in and started helping like productionize and scale the whole thing as an operation. And we've been off and running since then. It's a loose association of a tight web of trust of people that we've all known, that we can trust to be able to like help us with the whole project. So, I'll leave it there.

4:14 >> Yeah, and what exactly, you know, in detail here are you doing, you know, with those tokens that you're talking about? is it structured in a particular way? Are you looking in particular places? >> So, there's like there's like one it's it's mostly choose your own adventure because as an observation immediately what we're seeing is that people who have particular sub-domain expertise within the Bitcoin ecosystem on certain projects, are able to get very different results. And as an anecdote, after I found the first couple, I just talked to harness that would go through what I would view as a thing, like a process, and I made it very explicit like, "If I'm on a large machine, allow me to parallelize this.

4:54 We need to like I need to do this at scale." And that was like when I said I've scanned 100 repos, I wasn't individually going to each one. I just said, "Here's a list of URLs. Go." And GitHub and it started dynamically just like placing and like staging everything. and what's really interesting though is in those initial sweeps, I would I would pick I'm just going to pick a random thing of a thing a service and, or a kind of a type of software. And I would hit that type of software and it would come up like kind of middling. And then someone who's a deep domain expert would like look at the report and go in deeper. And then they would start nudging it a little bit more and it would escalate. You would start seeing that like it's almost as if I think all of the models had this knowledge the entire time, but the guardrails prevented them from being executed. And what happens with K3 is that it's just way more permissive to go do things. And there's a lot to be learned from that. I don't want to take up the entire microphone. But those were like a lot of the initial learnings that we're seeing is just very Yeah, it's been very revealing. I've there's been a lot moving very quickly.

5:58 Yeah. >> Sure. James, let let's get you in here. I'm curious, you know, sort of your view on on the role of the red team and and sort of the the purpose of going through all these repos. you know, particularly now. I guess the the better question here is is why is you know, why is this sort of effort happening now? is this something that could have been done weeks, even years ago? you know, I've got a tweet pulled up here where you know, you were looking at the Coldcard firmware looks like in in 2025.

6:29 so I'm just kind of curious, you know, what is this moment? Was this a big wake-up call and that's why this sort of effort's kicking off? Have people been doing this for a while? yeah. >> Yeah, I think people have been doing much more dilute versions of this for the past few months, but I think there are a few unique things about this moment in particular. Obviously, the Coldcard break is massive. And you know, when I was doing my audit back in 2025, I think the extent of LLM use was like, you know, as an exaggerated documentation helper or you know, it could generate a little bit of code for you, but it wasn't doing nearly the kind of wide-reaching agentic you know, almost autonomous work that we're kind of seeing now. and so I I think really indeed, what may have even catalyzed the Coldcard hack is the availability of these more sophisticated models that are, you know, less constrained.

7:32 and so I think the Coldcard Coldcard incident proved that conclusively. And so I think Rob basically, you know, very wisely ran with the realization that the frontier has changed. And at this point it's just essentially a race to not only deploy this more sophisticated and less restrained intelligence, but to actually build a a a purpose-fit harness that helps find this stuff because any any software engineer will tell you after interacting with these things, in a sense they're kind of dumb in the way that you get them to behave well and to yield useful results is you give them a very concrete fitness function or, you know, evaluation point. and so, you know, when you're searching for vulnerabilities in particular, coming up with proof of concepts and things like that, you know, you know, is paramount to actually have these things yield useful results. So, I think a lot of the the red team innovation is basically just, you know, Rob's work in devising a well-constructed harness, and then you know, not having any temerity about yeah, let's spend a couple thousand dollars on tokens and just rip through everything we can we can think to rip through.

8:49 >> Yeah, just one thing to expand upon that that's interesting is so, I built this initial thing in about 30 minutes or so. there are other members of red team who there aren't doing a lot of code work. They're just looking at the harness now and iterating on it cuz if you can imagine what I can find in 30 minutes, what you're able to see is that they'll basically take what I had as like a a little like toy car and like really like start thinking intelligently about it and then rescan again and you can find more things, right? So, I think from the initial like scope, I think the the reporting of things it's going to slow down a little bit as we kind of now look at processes and figuring out like how do we actually optimize this. I think there are definitely observed behaviors of how these tools work and trying to reflect over the past it's coming up on 3 days. It's been a very fast 3 days, but coming up on 3 days and realizing that this has to iterate to become better and better in general.

9:44 >> Yeah, I think this is interesting and the thing I want to kind of double tap on Rob that I think we we talked a little bit about, you know, earlier in the week, is this sort of trust issue and I think, you know, a lot of the work that you guys are doing is is hopefully sort of counteracting I think a lot of lost trust right now. but I'm curious, you know, if you were to make a pitch to someone who's sort of looking at this moment and saying, you know, how am I supposed to trust sort of self-custody, you know, how am I supposed to trust like the sort of code nature of of Bitcoin in this moment where, you know, LLMs are exceptionally powerful and and sort of, you know, also ungated, you know, with with the release of of Kami K3 in some sense.

10:28 what's your pitch to those people and what how is the work that you're doing sort of factoring into that? >> Well, I guess the start is that it's all about exercise of risk management, right? And concentrations of risk and risk distribution. Before I started AnchorWatch, for example, I was a big proponent of multi-vendor multi-sig because to be able to have a confluence of a single actor who could compromise a majority of your system that is like very uncorrelated is very low, right? Whereas what we're seeing here that we're single points of failure that we're trusting a default path within a hardware wallet and all of your eggs were in that one basket. And because of that, it's really, you know, we're seeing the fallout of it now.

11:10 I think it is fair to be skeptical, right? It is like I think that's a very fair and healthy, reasonable question to ask. What I would caution say that you don't want to throw the baby out with the bathwater that the Bitcoin experiment there must be self-custody, otherwise it has no value. >> Well, and what's interesting too is that we're in a kind of intermediate state where once this technology gets ingested and actually built into the development pipeline, there are going to be almost no secrets left in software, if that makes sense. So, you know, where you used to be able to kind of hide defects, now, I mean, as we saw, obviously with Coldcard, that's that's a phenomenon that's kind of going away.

12:03 >> Security by obscurity? >> Yeah. And you're you're you're going to I I think like in 2 years, you're going to have like like like bad software in a sense just isn't going to be able to exist in a context like this because you're going to, you know, anytime something gets gets released, you're going to have, you know, whatever three sophisticated agents pour over it with some of these harnesses. So, in some ways it it's like, you know, longer term, very, you know, encouraging. but, we're in this like kind of, very uncomfortable intermediate period where this technology is deployed unevenly. And so, I think what Robin and the right team guys are doing is is trying to close that gap as fast as possible.

12:48 >> I think it's just specifically too, since Bitcoin, the code is the money, the money is the code. This is obviously the first place you'd want to attack anyway. I think we're just kind of ahead of the curve realizing, this shift change, and what we're talking about over the coming months is just going to become routine practice in all enterprise cybersecurity, period. >> Yep. Yeah, and I think that's that's worth underscoring for folks listening to this, you know, whether it's on the policy side or or just, you know, the plebs, right? There's a an obvious downside to everything that just happened, and I think a ton of takeaways, and we're in this sort of, you know, accelerating phase of of, you know, intelligence, and that's going to come with problems and and solutions. and we'll get to this at the end of the podcast, but I think it it is worth circling back to the sort of open source versus closed source or or frontier lab sort of situation cuz that that does play into this as well. But James, I want to go to you quickly on on tripwire. can you walk us through what you're building there and >> >> It's fun.

13:51 >> And yeah, and sort of the purpose of that and and what the intended outcomes are. >> Yeah, so when funds started disappearing and when it was realized that the source of that was weak entropy on the part of the the cold card devices I was very curious about monitoring exactly, you know, who was immediately at risk, you know, who we could be expected to to be swept, what kind of attacks were ongoing and there's a a kind of a famous concept in computer security of the the honeypot.

14:26 And the idea with that is that you you place some vulnerability out there that you intend attackers to fall into so that you can gather information about them, you know, about the the nature of the vulnerability. and so I decided to basically deploy a number of vulnerable UTXOs that were kind of along a sliding scale of vulnerability in the sense that the UTXO that I deployed, you know, that had no added entropy that was just a default generation out of a Mark 3 cold card was swept basically immediately. I think it took an hour or something.

15:07 but if you add say two dice rolls, you know, none of those UTXOs have been swept yet. which which tells us actually that or indicates to me that probably the the initial attack on this stuff was pre-computed for some amount of time before they actually, you know, went and did it. maybe that's obvious, but so so we have you know, entropy all the way up to a few words of password >> James, let me pause you real quick. We have a slightly, you know, potentially less technical audience. Can you explain what you mean by pre-computed?

15:44 >> Yeah, so it's it's it's conceivable that the attacker discovered or knew about the vulnerability in the cold cards well prior to actually sweep going and sweeping the funds. what they they may have done is set up a program to run on some, you know, probably GPUs and basically just grind through the outstanding UTXO set to see, you know, what they could sweep and how much of the the search space they could explore before they actually started actually sweeping funds because the moment you start sweeping funds, it kind of tips the public off as to what's going on.

16:29 so we we, you know, we don't know when the attacker became aware of this defect. We don't know how long that they had been, you know, grinding on on this weak entropy prior to actually deploying the attack. so anyway, Tripwire ck.tripwire.com is is just my attempt, and it's an imperfect attempt in in the sense that because the attacker could have been pre-computing their attack, you know, for weeks, maybe even months, we we we really have no idea.

17:01 you know, just because one of these honeypots hasn't been swept at a certain, you know, number of additional dice rolls or number of additional passphrases words, that doesn't mean that that that, you know, class was safe in the initial set of attacks. so. you know, in a sense it's it's it's kind of a curiosity project just to map what kind of active attacks are going on, but ultimately, and maybe hopefully, the attacks that do happen, you know, we can harvest some information from in terms of where those UTXOs ultimately go.

17:36 >> Yep. Yeah, this is amazing. it's worth kind of looking into and you know, check out that website. It's ckerr.com, is that right? Sweet. okay, last question and then kind of I'll I'll turn it over to you and see if you have any questions. Rob, I'm curious, you know, for the red team, I I I assume you don't speak on their behalf, but you know, in in your sort of view of the world, is this an ongoing effort that should sort of get formalized beyond what it currently is? is there a sort of like diminishing returns to to the work that you're doing?

18:10 yeah, I'm curious like the the future of it. I you may not you may not know that far >> Well, I mean, the the diminishing returns is a no, because we've scanned a fraction of a fraction of a basis point of the open source software that exists out in the world. So, like definitely like it it's accelerating. Like this is going to accelerate, just the concept of what we're doing, not not even just like just the what we're doing right now as a idea, which really exists beyond what us as a group of Bitcoiners are doing.

18:38 I believe I've seen it now announced, so I feel more comfortable talking about it. OpenSats is starting an initiative to be able to specifically focus on this, which I think is incredibly important, especially going back to my observation earlier of when you combine these tools with people who have deep domain subject expertise in being able to deconstruct what is happening. It is a very natural fit. Additionally, unlike maybe other cybersecurity firms that are doing pen testing, this is entirely isolated on your local machine. You're not attacking any infrastructure. You're just spinning up code on your local computer, open source software, and basically battling it out to see what happens. So, it's you know, much cleaner that way to be able to do that. And if you combine those ideas and learnings with people who are domain experts, I think you're going to get even better. And over time, there's going to it's going to get a lot more efficient and even better. Because, like I said, I think I think I scanned with my tool almost 200 repos myself at this point cuz I wasn't doing anything. All I was doing was saying, "Here's a list of URLs." And go. Like, I didn't And just mass pushing it out like into like the documentation side of things.

19:48 it's That's step one. And so, I've actually at this point, as of like this afternoon, I'm taking like a little bit of a break on scanning things. And I'm now looking at the internal operational processes of how to make it better from learnings that we've gone up to this point. And I think that's just going to compound. >> Fantastic. Ken, let me let's flip it over to you here. any anything you want to jump in here on?

20:11 >> I think just to one earlier point that was addressed. I mean, I I know some folks who know a lot about this. You could see a world where 3 years from now, five certainly five years from now, where the only actors that can conduct CNE, you know, cyber network cyber network exploitation are either nation states or really, really well-funded actors. Cuz it's just going to be too expensive. Cuz everybody's going to have access to decent models and the ability to to to defeat those is just going to require a nation-state level capabilities. I think that the the thing that we're concerned about, we need to be concerned about now, is the immediate future of open source. Cuz open source is always in some level under threat in in the policy circles.

20:45 It just is. It's just too easy. it's it's an easy target. So, there's always people nipping at the edges of how do you restrict open source? We see it frankly in in the in the clarity bill. how do how do we What do we need to do? And I'm this is a rhetorical question. Maybe it was an answer. Maybe you guys have something to say about that. What do we need to do with the interim to make sure that open source developers, the community has the resources it needs to survive this. Cuz I I do think on some level I I I talked to a developer this morning who runs a well-known platform and he he he said this is existential for open source. And I think he's battle-scarred and he's, you know, probably traumatized by what he's experiencing right now. But on some level he said it, this is almost existential. I asked him a question, like for example, if you would would open source developers KYC for good models?

21:29 He said, "Well, if they don't, they're, you know, they won't exist, you know, they'll be gone in a few in you know, in a few months." So, either the ones that won't won't survive, the ones that will will survive and there'll be compromises. So, I I think this is what we need to think through exactly at PPI, like what do we need to do to make sure that open source survives this intact, sort of at least resembling on some level the the the the freedoms developers had up until this point.

21:53 >> I made a tweet on June 9th saying, "We're going to replay the entire debate of the 1990s on making math illegal, aren't we? Last time it was prime numbers and now it's going to be linear algebra." I don't actually understand in a free society how you're able to regulate this. Just fundamentally, like you're saying I can't have like large matrices on my computer? I can't have a bunch of a bunch of floating points on my computer?

22:15 Like what is like what kind of country are you living in if you can't have math on your computer? And we had this play out in the '90s with RSA encryption. Like I'm I I'm sorry for coming off aggressive on that. I just actually don't understand with the principles and values of America as an experiment how you can actually Like what what is the restriction? Like if I download the model like is it going to be 6102 or is it just like there's going to be a stopping of this? I don't know. Like math and information just naturally has a way of becoming free and I think we have to kind of rethink a lot of just foundational policy things from like the core values of our constitution and the republic about making sure that we're not sacrificing those principles to be able to be in a panic. I I just don't know I just I'm sorry I don't mean to be so >> >> I don't mean to be so direct and blunt about it. But like I just don't understand.

23:03 >> Well, and and and Ken, correct me if I'm wrong here, but I I think you were even getting at a more mundane point, which is like, if you're maintaining an open source project and you can't afford you know, the thousands of dollars in tokens to to scan your project, you know, on an ongoing basis, are you even going to be able to to kind of survive? setting aside whether or not you can get access to to the good models.

23:28 >> Well, maybe for some context, it doesn't cost thousands of dollars to scan your repo with my tool. Cuz if it did cost thousands of dollars and I scanned hundreds of repos, I'd be millions of dollars to spend right now, right? So, start realizing like the amortization of cost. I think that I was doing and it depends on the complexity of your code base, but some of them I was doing for under 10 bucks and some of them would take like 60 bucks, right?

23:47 Like just for This is not like you need to have an entire grand foundation being able to fund you to do this. It's cuz of the scale we're doing at the moment. I think that's just an important thing to call out of the asymmetry of this, an unfortunate asymmetry which is emerging. And I think from a policy perspective, what I would caution is that Kimi K3 may have went open source last Monday. China and the labs that are working on it over there had it for weeks if not months earlier.

24:11 So, did that I think that's the actual like realization here of something to consider. >> Yeah, I agree with you. I don't I don't think the Chinese will necessarily want Kimi K5 out in the wild, right? Until they've until they have K6. >> They may start locking things down, which will be even worse for America. >> And that would >> That would unfortunately and tragically make things worse for white hat and you know, people that are trying to do the best they can.

24:36 >> Yeah, let's double tap on this. I think we sort of briefly touched on it earlier this week, Rob, but let's get into the substance here. Kimi K3 has been sort of instrumental in these efforts. Is that correct? >> Instrumental, yes. That is that is the inflection point here of what's changing everything. I think I mentioned it last time like Kimia is just the one that goes in. It's Kimia is the one who breaks into the house and finds where things are going.

25:03 That's what it is. All of the other ones are like, you know, boy scouts. So, they don't want they don't want to do anything. And then Kimia K3 just like like pops everything open. >> Yeah, I love that. >> It's the best metaphor and having spent personally thousands of dollars at this point looking at seeing the mass outputs of this, that's how I describe it. >> Yeah. And the theory is and it's probably certainly true that, these frontier models, Fable, GPT 5.6, they could all do what what K3 is doing, but they're they're blocked from doing that.

25:34 >> For sure. Like and there's multiple like, when I mentioned the the initial anecdote was with looking at the cold card firmware saying that there is a bug we need to go find it. Like, I got instantly downgraded on Fable. >> Right. >> It knows it knows the answer. That's why it's downgrading me cuz it's looking at what it's going to send me be like, "Ooh, spicy. Can't have them have that one." Right? Like they know the information. It's not a secret.

25:54 >> I love it. Well, let's let's wrap up here. I want to give you guys back some time. I think, you know, question to both of you to wrap up here is, you know, Kimia K3 open source open weight AI has been absolutely instrumental to, both sort of pinpointing the issue that that arose with the cold card breach and also sort of scanning and and doing the, you know, sort of recon on other open source repos, or or code bases.

26:26 you know, if you're if you're sort of platformed in front of a US policy maker who's, you know, should we regulate, open source open weight AI, should we put restrictions on it, should we ban it outright? What's your pitch to those people, on on why that's just, you know, wrong? Rob, I appreciate, you know, the, the sort of free society, you know, argument and I think that's spot on. I'm curious from a more like pragmatic perspective. Like if you were to to say these are the things that that will not be able to happen if we limit Kimmy K3 in the United States.

27:00 you know, I think that that will also be a compelling piece to share with some folks. >> It's a very awkward position for me being someone who is not deep in instantiate like deep in the policy world to understand the framing of this. So just allow me for my own perspective and I I made the point earlier. >> And to be clear Rob, like this is the point, right? Is like, you know, Ken and I were the policy people, Zach were the policy people. Like policy people also need to hear from builders of like what are the implications of their decisions around policy on these issues. That that's all this is.

27:36 >> just think it's a dystopian society if you ban math. >> Right. Like you you know what? You could have gotten away with it with the original paying attention, whatever attention is what matters, whatever paper that Google came out with with the tensor originally. You may have been able to black box it that and just like disappear it, right? Like I understand like with very critical things with weapons and stuff like, you know, you just something feels of research just die off because they're too important to kind of be out in the open academic world.

28:05 Like like Los Alamos, like you just have to necessarily just like not be talking about this at every random university. I don't know at this point how you undo that. So accepting that you can't stop it, the information in the knowledge graph here of what's there. It would not Maybe it's frontier research in theory is like locked off, but like even then like you just need one bad actor to leak it to another like nation state and then you're just have you're back to where exactly we started right now. Like I think it's I don't understand with keeping the values of the American experiment and being able to reconcile banning of math, how you're going to be able to do anything that is actually going to work and actually protect Americans. I don't.

28:45 Like and I think also too about the open source piece, James made the point earlier, security through obscurity is dead. And just because your code base is closed source, doesn't mean that the open source players aren't doing more active things like pen testing, trying to get into your system, rapidly trying to escalate permissions and privileges. Like, I am not convinced that just because you keep your code closed source that you're going to be safe forever because this is that is only relevant for the very narrow kind of security research we are doing at this moment with the red team project. Of looking at open source code and battle testing it, it it's going to take many different forms very quick.

29:25 >> Yep. Anything to add there, James? >> I think people need to acknowledge that the reality is that we're in fifth gen warfare and you know, I'm I'm sure all of you guys saw headlines about attacks on water systems, attacks on hedge funds, you know, very likely made possible by the kinds of AI that we're now trying to familiarize ourselves with from a defensive standpoint. And I think when you're in a war and you're doing anything to technological innovation and development, you're you're you're putting yourself and your civilization at a significant disadvantage.

30:06 And so I think yeah, policy makers would be very you know, you know, what if when we during the development of the internet we had we'd clamped down on, you know, computer ownership and you know, had had to have licensure or something to get to get on the internet. I it just the the, you know, our our grasp and our mastery of that technology wouldn't be nearly what it is today. And so I I think it's America, you can't ban math, and also it would be a stupid idea, too.

30:34 >> I love it. Great message to end on. Thank you both for joining us. keep up the good work and yeah, we'll we'll keep cheering on from the sideline here at BPI. >> Yeah, thanks guys. >> Thank you. >> Thank you. >> Our pleasure.