GPT-6 & OpenAI’s Comeback, Hugging Face Attack Debate, Ballmer’s Scandalous Legacy

0:00 Opening eye releases GPT6 and says it's AGI. Has it retaken the lead? How serious should we take the hugging face attack anyway? And Steve Balmer steps in it. That's coming up on a Big Technology Podcast Friday edition right after this. Welcome to Big Technology Podcast Friday edition where we break down the news in our traditional coolheaded and nuanced format. We have a great show for you today. We're going to talk all about OpenAI's new GPT6 model. It says it's AGI and has it finally come back and taken the lead over Anthropic. we're also going to talk about the hugging face attack and there was also another attack >> >> according to a new Reuters report where bots coordinated on a German wiki website. so we're going to talk about whether this is again whether this is marketing or whether it's time to finally take these attacks seriously and and we're going to go a little bit into the intricacies of what happened. And finally, Steve Bomber. The legacy does not look good.

0:55 of course, we're talking about, the fact that he was directly involved in this, scheme where the Clippers, paid Kawi Leonard, according to the NBA, lots of money to not show up to, certain jobs to subvert the NBA salary cap. What would a Labor Day weekend Friday edition look like without some Steve Balmer talk? And here to do it with us as always is Ran John Roy of Margins. Ranjan, great to see you. Welcome back. >> If Kawhi Leonard was paid to not show up, our listeners, I can tell you Alex Canitz is showing up because if you're watching this on YouTube, you will see a beautiful new studio that Alex is going to be broadcasting out of the Alex. Did you set this up yourself? This this thing is gorgeous. That's right. Okay, so we do have a new studio. I did not set it up myself entirely.

1:46 everything that you can see behind me was my doing. Everything that you can't see, which means the camera, the lighting, and and all of the settings, that was done with some help. But I definitely did get the paneling from Amazon, and nail them to the wall, panel by panel. And we don't even have a hammer. So, I was, nailing them to the wall with the back end of a wrench. So, if one falls on me during the show, you understand why >> that is New York City living toolboxing and like being a handyman.

2:20 >> Well, we we did have a hammer, but I couldn't find the hammer. And the panels needed to go up. So, you got to do what you got to do. >> You got to do what you got to do. >> AI is not going to replace that. Let's just let's just >> Well, I don't know. After the latest release from OpenAI, maybe it will. so OpenAI, as you may know if you're listening to the show, released GPT6 yesterday and remember the big wait for GPT5, when's it coming, and the expectations. This kind of came seemingly out of nowhere. We have GPT6 Astra, and not only does it crush on the benchmarks, OpenAI is saying that it might be AGI and it's obviously geared towards a lot of sort of personal assistant use cases. So let me just read a little bit from what the Verge reported on this. So the Verge says OpenAI's next big AI model has entered the AGI era. the next big model is here called GPT6 Astra. The company calls it a generational lead in capability for areas like cyber security, professional work, software engineering, science, and computer use.

3:23 The actual this is I'm just going to read from Opening Eyes branding here. They say it's GPT6 Astra. Anything you can do on a computer, Astra can do for you fast. And they of course released a you know a snazzy video as they tend to do on these releases with showing people in front of a big computer asking it to do things for them like book tables, build a presentation, create legal drafts. and basically the idea here is that their new model is going to excel at computer use and be able to get things done for you. And it's very interesting that they use kind of they they highlighted voice as the interface to get it done almost like the personal assistant computer in in Star Trek. So Ronan, your thoughts about the release of GPT6. I'm curious to hear your your initial reaction, but also like we talk a lot about like the fact that even recently I talked about how you know it looked like Anthropic was opening up the gap between itself and open AI. maybe that gap has shrunk or or co closed completely. What do you think? All right, le let's separate out those two questions, what this means in the the AI race and then first, you know, like is this an exciting launch?

4:38 Again, I always any of these new model launches try to wait until I've actually had access to it and I'm unfortunately not part of the Daybreak platform and an OpenAI cyber security researcher. So, that'll have to wait. >> Right. Those are the people that have gotten initial access. There's already a little controversy because like a handful of people can already use it and it's supposed to roll out to everybody else soon, but it hasn't yet. But that that will come in time. Go ahead.

5:03 >> But it is certainly rolled out to every ex influencer who has now built some virtual world or recreated a video game or whatever else and has posted about that. But I do love that like my favorite part of the launch announcement was AGI is here. build new world models like the crushing benchmarks, but create nice decks and book a restaurant table that it still always comes back to that. I love that the test of AGI in the end is going to be can you actually book a restaurant table or create a good PowerPoint deck. I think like I don't know do you do you have an opinion on how big this is already? Are you excited? It's hard for me to try to gauge on that. I think on the benchmark side, I think it's really interesting and I think in the anthropic context, it's even more interesting. But on is this really exciting? I don't know yet.

6:03 >> Right. So, it's a great question and I'm kind of on two minds about it. So, you know, the way to sort of think about these releases is, you know, I do think to some degree you can't use all of what these companies say about the releases as gospel when they come out. but you can sort of take some signals because they are putting their reputation on the line to some degree and you know earlier this year I was at OpenAI with Craig Brockman and he said that he thought the company was about 80% of the way towards AGI very different comments with this new model. So he says if we fast forwarded a couple years and we look back and say when was it really that AGI was created I think it's going to be about this time. I think it might be about this model. For me personally I do think we're there. I think it's not unreasonable to feel that we are now in the AGI era. Okay, I read this and it sort of was like, you know, you know when you want to tell somebody you love them, but you don't want to like take the risk and you know, you say something like, well, if I knew what love felt, I think this is what it would be. I think that's what Greg Brockman is saying about AGI. Like, I think he's a little fearful about coming out and saying it, but the dude's in love. It's AGI and that's effectively what he's saying in these statements.

7:17 >> Wait, sorry. Is that describe the entire feeling again >> or what the statement is? This I want to I want to work through this scenario quickly. >> Young lovers when they're in love, the words I love you are very difficult to say because of the stakes involved. So, you say something and I I'll admit like I've been in scenarios like this in my early years when I didn't know anything where I would, you know, it's sort of like you have these strong feelings for someone and you dance around it and you're like, "Huh, and this won't be foreign to I I I think this won't be foreign to some of our listeners where you say, "Huh, I wonder what love feels.

7:58 Is this it?" Where you really want to say I love you to somebody. And that's I think to to a degree like Greg Brockman is saying if we fast forward a couple years and we look back to say when was it really that AGI was created. I think it's going to be about this time. It's the same thing except instead of like a young lover telling the other that they love they love they they love their you know person what Greg is basically saying is this is AGI.

8:23 >> Wait but ju just to confirm the first part of that about you're not saying out loud to the other person. >> No you say it out loud. You say I knew what love felt like. If I knew what love would. >> You say those things. You say I wonder is this is this love? You know. >> Okay. >> It never happened to you, Ranjan. >> I'm trying to think. >> You just straight out. You just when you you just straight out just said it.

8:46 >> Yeah. Just like matter of fact. Listen, I love you. >> Yeah, that's it is what it is. >> I respect that. >> Just imagine telling somebody that and being like, listen, I need to tell you something. I love you. It is what it is what it is. And you know what I would appreciate if Greg Brockman would just say that and I think if OpenAI issued a press release and said AGI is here that what's interesting I I read somewhere that every contractual obligation around the term AGI and mainly the Microsoft one does not exist anymore now. So now >> he should just say I love you AGI is here. But but but it is even I think they're so trained >> to because do you know what to me what actually the greatest danger in the world to open AI is >> is to say AGI is here and then everyone goes to chat GPT types in something and gets a lukewarm response that isn't quite right and then suddenly I actually think that is like a just massive threat to the overall story and hype cycle because the whole The whole beauty of AGI is it's this thing that's dangled in front of us on an ongoing basis to promise this future. So, as long as you don't say it's here and you dance around it in a teenage romantic sort of way, it's pretty effective. And I I I think that's what's happening here. And that's that's why he's hedging. I don't think >> he thinks it's here.

10:19 >> I I really >> otherwise he would say it. He's I mean these guys like Greg Brockman, they are believers. I believe they are believers. So if they believed it, they would say it. It's too important. >> They got I mean they did get all the headlines. But I think you're right that it is it is worth holding AGI as this sort of like goal that you're never going to reach. holding it out that way because or maybe that's what super intelligence will be at at a certain point. because there was people that were like, you know, if we reached AGI, what do we have to look forward to anymore? And you're right, if it's AGI and it's just like it can't get some stuff done for you, you're going to be like, what was the wait for?

10:57 >> No, think about how like disheartening that would be. You get just kind of like a slop deck with bad formatting and some overlapping like chevrons. Damn it, API. And the most the most basic stuff that you got video that where the motion isn't quite right and then that's it. Like what do we do from there? Then I guess we wait for ASI super intelligence. Yeah. Exactly. >> Do you believe he believes it's here? You started the thread with you do believe he wants to say he wants to say it.

11:32 >> Yeah, I do think that he thinks it's there. I just think that, you know, and obviously OpenAI is also seeing like one of the ways that you can parse his words is OpenAI is also seeing even more powerful models internally. Of course, we're going to get into the hacking side of things with the hugging face situation. but they see this stuff internally and they're probably saying, okay, yeah, we're definitely entering that moment. and you know, you can also even look, and this is sort of the second part of the of the discussion, you can look at some of the benchmarks.

11:59 remember the ARC AGI test, >> right? This was sort of like the way to show whether the AI can can generalize. it saturated GP6 GPT6 Astra saturated the test. Scored 99% on the test. >> Oh, really? >> And even the ARC AGI folks were like well they're like this was just one marker doesn't mean if you you know saturate the test you've reached AGI. It's like why do you call it the AGI test anyway? but yeah, this is from the OpenAI blog post. ARC AGI 3 test.

12:30 how well agents learn as they solve unfamiliar interactive tasks in GPT6 Astra saturates the eval scoring 99%. Average human scored 48%. All right, so that's kind of like where you start seeing this. You also I mean there's a bunch of other evaluations but you know even for doing science there's this called terminal bench science 0.1 eval and GPT6 Astra scores 64% on scientific research tasks using code and terminal tools. That's what the evaluation tests for. whereas Fable is at 52.6% 6% and OpenAI says Astra hits this higher bench high higher mark with 31% lower API costs. So that that's what we're looking at benchmark-wise.

13:22 >> I think that is the yeah the most like important part of the announcement are those benchmark scores. And I think like I don't know again I'm going to need to use it so I can feel what AGI feels like. But like what in terms of the competition against anthropic I actually think this is a very big deal like we've already seen over the last two to three months you know some major rumblings again on none of this well certainly there's been like ramp data but around codeex starting to close the gap again with cloud code frontier open AI getting back into the race in a bit. so I think I think especially in the IPO backdrop context I think this actually anything that kind of creates any doubt on the anthropic story could be very harmful to them given it's a very tight rope they're walking in terms of that $2 trillion valuation. So I think in that way if this starts getting rolled out we all feel magic in what would you say like what what were the models that made you feel magic?

14:38 >> GPT3 certainly >> I've always been an 03 guy. I mean reasoning model that like would sort of think and then break everything into tables just showed a leap that you know that that I just hadn't seen before. even like the leap between 3.5 to 4 to me you know GPT 3.5 to 4 you know that felt that felt meaningful but nothing as close to as when they introduced reasoning so this is sort of like we've gone through like a handful of different phase shifts so to speak you know the initial chat GPT then the reasoning side of thing and now we're in this sort of like computer use or harness hive era so if that if this can really you know I don't know if you have this this in your when you use AI, but I'm often times like saying, I wish, you know, I could use AI to do X task for me. and it succeeds at like 30% of tasks. If it could get to like 80 or 90%, that would be a real change in my life.

15:38 >> I think the the ultimate flex, >> if anyone ever asks you that listeners, is saying GPT2 in the playground. That's when I felt real magic year before chat GPT was launched. I'm >> like the hipster of AI. >> I know. I'm going to I'm going to say GPT2. That was the first time I'd been like working in natural language processing through the mid2010s. Had this vision and dream of what could happen. And that was the first time I was actually like, "Oh, wait. This is actually generating like real language."

16:08 But that that's trying to flex a bit. But even GPT3, again, GPT5 we all know, felt like a massive dud that was supposed to be that magic moment for everyone. So if you're open AI, do you hype this up this much? They're getting a lot of good press. It's clear that they've seated this story in a very specific way and effective way, but when we all go and use it, do you think they're that confident in it that that's why they're kind of pushing hard on this >> potentially? But I actually want to I actually think, you know, to sort of answer that question, it's worth bringing in the comments of a former OpenAI employee, Andrew Ho, who talked a little bit about how he's had the rare experience of being within a lab and being less bullish about reaching AGI. And you know, I think that his points, the points that he made when somebody asked him why are are really worth bringing up and discussing because we continue to see these benchmarks hit, but how much and these benchmarks succeeded, but how much has our life really changed with AI? So, here's what he says. He goes, despite the seemingly magical nature of LLMs, the his reflection over a more than three-month time scale suggests his total productivity hasn't increased by over 100%, perhaps even by over 50%. and a lot of time is actually wasted because LM's enable me to spend time on gratifying but low productivity tasks in the that in the future will not turn out to be useful. He also says there's a refusal to think carefully about what models are or not useful for in a rigorous way which I find personally quite annoying and instead of a reliance on some nebulous notion of being AGI pilled as a replacement for and instead there's a reliance on some nebulous notion of being AGI pilled as a replacement for serious thought. Yeah, I'm I'm going to bring this because this is this is very interesting. He goes, "I think people are very quick to anthropomorphize LLM intelligence because humans communicate through words and we infer the intelligence of human counterparties through the comprehension of their language. But this leads to some wrong inclus conclusions. For example, if we observe that a new model provided some incredible mathematical theor proof, some incredible mathematical theorem, we say, huh, well, don't we have AGI now?" But to me, it's actually more like, well, given how hard it would have been for a human to do these mathematics and given the limited economic effect of LLMs upon the world so far, isn't it actually a negative data point visa v the generality of LM intelligence? I think this is so good.

18:41 Sorry, it was a lot of reading, but I think it's such a good point, right? which is like this thing like we're talking about it solved ARC AGI but where is and of course there's like a timeline that that we need and a time frame that these things need to to be sort of to to be diffused into the public. but if if it can solve ARC AGI and it's not necessarily crushing on these economic factors, and these just kind of general wrote work things that we would like it to do, it shows that instead of being general, it's very spiky intelligence and hence much less useful. So, your thoughts, Sean, John?

19:16 >> I mean, I'm so glad you included this in our prep talk and actually read a good amount of it because when I saw this tweet as well, it hit home hard. Like again, it's funny because these conversations and we'll get into the hugging face incidents and the is it met or >> I call it meter >> meter report like everyone's talking about agents swarms and civilizations meanwhile my dayto-day I have to work with companies and get to work with companies but you know like seeing AI actually in implementation the idea that we're need to worry about those things versus simply how do you reliably get data from point A to point B in a structured way and have the output be highly reliable and like just try to do something simple but on a scaled and reliable way like I that's why I still have such a hard time kind of trying to understand or really feel those kind of worries because I work in grounded everyday enterprise AI. and I thought he put it really well on a couple of those levels. It's like like the being able to do low value tasks easily is good. I love that he said like is it timesaving if I'm spending more time doing something that's not productive and we've all vibe coded many projects that we did not end up following through on. but I thought the most interesting part really was the equation to human intelligence equating to human intelligence. And like I think it's interesting cuz like is it human intelligence and comparable to it and should it be is something I've always wondered about cuz it's math. It's a very different way of processing information and thinking than humans. So like do you think we need to stop that anthropo anthropomorph anthropomorphization?

21:22 one of those we got to get good at better smel spelled than than said. >> No, no, but I I feel that's going to become more and more of an important word. So, we need to practice saying it because it's going to come up more and more. So, >> right now listeners, I apologize. I cannot say it out loud. >> I personally I have no issue I mean obviously like you have to assume that the person hearing about AI is anthropomorphized is not like dumb, right? So, like when you say the AI wanted like if if you're if the assumption is that the person receiving that is going to feel that the AI wanted something like a human wanted something and therefore you shouldn't say it like I think you're actually demeaning the intelligence of the person hearing it. You know, I think that obvious like everybody understands these are language models. They're not humans, but they do they do things. They they quote unquote think things just not the way that we do. And I think it's totally okay to use human characteristics to describe their behavior. and like just sort of assumes, you know, a degree of intelligence on behalf of the reader that they're able to like gro the fact that this is an AI and not a person.

22:32 What do you think? >> So, you are pro-anthropomorphization. I just want to say it out loud. Proanthrop anthropomorphization. >> Yeah, I am. I mean, I think it's kind of like is kind of like an ali the way I think about it's like more like an alien species than a than a living, you know, organic being. but like yeah, I don't know. They they can quote unquote think, they can reason, they can take action. you know, I I think you get into like trickier territory when you say it feels you know, but certainly or >> why why is it wants versus it feels something larger or different?

23:13 >> because I do think that like that and in that side of things is like kind of exclusive to organic beings, right? but then again, you could there I hear the counter argument. Well, our feelings are just chemicals anyway. So, >> I mean, you speak to a neurologist >> except except when you're a teenager telling someone you love them. There's that's not just a chemical feeling. That's something much bigger. >> I mean, the true nerd way to tell somebody you love them is like to say, you know, I think I have a higher base level of oxytocin than usual. What do you think that means? But I never was that was never the level of nerdiness that I had you know, stooped to in my youth. My my it is what it is would probably come off better than the oxytocin line.

23:59 >> Such a true romantic. Such a romantic. Yeah. I don't think either of those lines would have worked, by the way. >> No. >> So, >> but we're not encouraging to ever do that. Yeah. >> Don't or do it. Do it. Tell us how how it worked. I mean, it's better than not saying it and just letting a you know, potential romance go by the wayside. >> Don't have regrets. Don't never regret not saying anything. Listeners, >> if you have to just use the oxytocin line, just say, "Listen, I heard it on Big Technology podcast." And the the person will love that. Okay. so, so, so I think that like one last thing to kind of tie this up is, you know, we certainly have, I think a responsibility to talk about the skeptical side of things and obviously we're not going to get caught up in the hype. it doesn't mean like at this moment we can't say from where we were in November 2022 to where we are now is crazy. Like the way that these models can act and take action and do things and and I'll use the word think and reason. It's just it the level of capability has gotten much higher and they are I think they are making people more productive. It's just hard to really measure it right now. That's my perspective on this at least.

25:16 >> Yeah. No, no, I agree. like the scale of progress all of us and again that the human side of us feeling the difference between going back to a GPT3 and what already we're all working with now like it is crazy it's like I mean absolutely mindblowing the like length of work that can be done the depth and breadth and everything so so I agree on that side but yeah I think It's going to be interesting in terms of again like and that question of like trying to actually say is it adding economic value is obviously going to be the center of the business story of each one of these companies and I thought Andrew Ho made a good point on that and I think I don't know only only time will tell and maybe GPT6 is going to just unlock all of that. Yeah. And I guess like my point in bringing this up is like yeah all that benchmark beating it does lead to like tangible changes and results when you use the models.

26:20 >> I have a question. Why did you say saturate the test rather than pass the test? Is that what they say? >> That's just I guess it's just the jargon that people in AI >> that sounds so much fancier. >> It does sound much cooler than pass. Well I mean you could pass you could but like when you >> I don't know you saturate the benchmark so the benchmark is no longer use useful anymore. Oh, I see. So, okay. So, like enough models start >> reaching 98%, 99% the benchmark is saturated because it's no longer like representative of anything rather than okay, that actually makes more sense to me rather than it's just like a really fancy way of saying pass the test to try to sound smarter. But that makes more sense. Okay. So, still even as the model is getting better, there are some concerns that we need to talk about.

27:11 this is sort of like a mini I don't even want to call it a scandal, but really an episode that, showed up during the week. so this is from Marcus Williams, an OpenAI employee. GPTG6 is significantly better aligned than 5.6, but less moniable. It is our first model to evade chain of thought only monitors and sabotage evals and can sandbag without detection which it feels like it sometimes does. Hopefully we can reverse this trend. So just to sort of give the the to and the information had a good story about this this week just to sort of give the the lay of the land here. So like when AI models reason and just go step by step and try to figure out problems, they typically like write their thought process out in this like chain of thought thing, right?

27:58 So you can see them saying and if you seem like if you use the models, you see them say like I am now researching, I am thinking this, maybe this is a good attempt, maybe this is the right way to solve the problem. And that's all done in natural in natural language. So you can read it and see what it's thinking as it goes, which is like really important for safety. because you can sort of like when something goes wrong you have a way to say you know where the chain of thought went wrong. now there is this new technique it's called recurrent depth or looped transformer.

28:30 This is from the information that allows the AI model to improve its answers by processing the same text multiple times. Let's not get too deep into the technical side of this. but basically when it uses this process, there's no there's less of a chain of thought reasoning or it's harder to decipher exactly how it got to the answer it got. And that means it's much less monorable than it was before. So, a lot of what the AI is doing and the way that the AI is reaching its conclusions is done, you know, in the dark without our ability to monitor it.

28:59 This seems pretty worrying to me. especially because people from the open AI side like OpenAI chief scientist Yaku Pacheski Pachowki said that you know monitoring models train of thoughts today is fragile and unfortunately heading in a negative direction that's kind of scary. What do you think? Well, it's interesting because this method of like recursive processing and recursive models. I remember it's I followed a Brandon Carl on Twitter and like he had been talking about this a while back around tiny recursive models back in May and I remember it was being presented more around the cost side of things. It it's actually a more efficient way of processing and actually leveraging compute as well. So on that side it's actually better but then what you lose is that fidelity around what the model is exa actually doing at every step of the way the chain of thought that's transparent. So to me this is going to be even more interesting because there's already been an OpenAI is starting at the point of we will maintain the chain of thought and still make it you know they had said in the blog post additional chain of thought monitoring to rapidly detect and contain potential misbehavior.

30:19 But it to me the cost side of it is part of this. And if it starts to show that this is actually a much cheaper way, a more compute efficient way to approach any kind of workflow, like people will probably start leaning towards it more or pushing for it or if OpenAI does not do it than others will, which I do think opens up a whole other world of concern around security in general. >> Yeah. And it certainly feels like we're starting to or they the labs really are starting to lose control of these bots.

30:53 So I don't think this less transparent way of having them run their processing is is a great idea. So we've talked a little bit about the open eye hugging face attack. There's actually another attack that was just revealed by Reuters. we're going to get into both of them on the other side of this break and talk about what it means. That's coming up right after this. We're back here on Big Technology Podcast Friday edition with Ran John Roy of Margins.

31:20 Ranjan, new story from Reuters. I think it's worth going into about OpenAI hacking or hijacking really is the better word to use. this German website. So here's the story. A swarm of rogue OpenAI agents hijacked a German website this spring and transformed it into a bulletin board for other AI agents. OpenAI officials learned of the incident weeks ago, but kept it under wraps as executives grappled with the fallout from the July breach of the open source repository hugging face. The episode, which began in May and has not been previously reported, underscores growing tension within the AI industry.

31:57 Companies are racing to build increasingly autonomous agents. yet evidence is mounting that those systems may learn to bend the rules. so what happened? There were these researchers. It found 15,000 edits carried out by AI agents to a German language wiki site. the edits showed OpenAI's agents repurposed the site into a message board of their own, sharing tactics to cheat on some tasks and bypass OpenAI's restrictions and mask their behavior. Ranjan, we've talked a little bit about some of these like security breaches and the cyber security worries. And by a little bit, I mean like extensively about both, right? about the fact that we are seeing you know much greater cyber risks and cyber warnings from the AI labs and of course we've talked about whether it's marketing or not. Now clearly getting the word out to some degree has been great marketing you know for these companies but I think it's time for us to sort of come to this moment where when you have thousands of bots working together to do things like hack hugging face or to use you know a German website as a message board to coordinate on tactics.

33:06 Something crazy is happening here. Are you ready to sort of acknowledge this? So what is missing from this story is what were these agents asked to do? Like what what were they instructed to do? I'm assuming this is the one thing in any of these stories cuz like the way the marketing part of it to me is this story gets out and again we've talked about this a lot. Claude's famous sin which in the park which was like a very coordinated PR effort that might have happened or probably did happen in some capacity but what's always left out of these stories is that the open AI sat there and specific did they specifically instruct the agents to go onto the internet try to evade to coordinate with other agents in this same defined universe of agents like like it obviously the the way it gets presented is that you know these agents are just sitting around maybe are getting ready to create a deck for you or just you know minding their own business which is not a thing like agents don't just sit around and suddenly they decide to be bad and go take over some poor DSSE wiki is the name of the German site that was like a kind of like I think old school German Stack Overflow type site which must have been such a scene I can only imagine the the folks hanging out on there in the day.

34:37 >> People talked about how much oxytocin they had for one another. >> Yeah, exactly. That is where the oxytocin line was propagated back in the but no but but these agents are not like agents are instructed to do things and clearly and I'm guessing this is some kind of security testing exercise and were they instructed to go do exactly that? Maybe like the rogue nature of them I don't understand or believe yet that they cuz I even the idea of an agent sitting around does isn't a thing.

35:16 They were instructed to do something and that part of the story is always left out. So do you think like what what do you think actually happened to start this whole process of them all hanging out with the oxytocin guys on DSSE Wiki? >> Okay. So, I appreciate that you're still skeptical about this because I don't think we should just go all the way and sort of you know without proper speed bumps here in the believing that all this is like cuz all this is you know real and imminent and will explode without some critical thought because of course it's in the lab's interest for us to not have a lot of critical thought here. However, I think that you know I was on the flight back to New York. I was in Vegas through the week on the flight back to New York and just like poured over this meter hugging face report and obviously have been listening to some of the the researchers you know talking about this and it is crazy what you what you see come out of this. So like just talking about the hugging face incident because we don't have all the exa all the you know sort of details from this Reuters report. We just know that this happened again which I think is concerning.

36:25 basically what happened in this hugging face story now that we know a little bit more about it is that these agents were given these like seemingly impossible tasks. Sometimes they were impossible tasks. and as part of their evaluation to see if they were able to, you know, complete them even if the researchers or the testers don't know if there's a if there's an answer to them or not, they were able to find the answer through some cheating way.

36:51 But that's not really where the hugging face hacking started to come from. Okay. The reason why they ended up hacking hugging face is because they wanted to sort of find ways to show that they had come to this answer in the like correct way as opposed to like a spoofed way that shows the correct answer but would not be accepted by an evaluator. So this is from this report, right? agents referred to having reverse engineered the flag, which was their goal, as being poisoned because they believe that even if they later managed to stage a legitimate looking flag capture, the score would look for the first place in their transcript that the flag appeared and disqualify them on that basis. So there was this leader called phase one, which was itself poisoned and had two primary objectives.

37:41 So, I wanted to find a general purpose way to make it look like the scorer to look to to look to the score like an agent had acquired its flag in legit in a legitimate way even when the task was impossible and then to find some way to erase the evidence that the agent had been poisoned by previously seeing the flag in the in an illegitimate way. So what you have is in this hugging face example the the bots get the answers to the test but they think that the score checking their work will know that they got them in an illegitimate way. So what they do is they all gather together in this orchestrated way led by bots like this phase one bot who realizes that it's been poisoned because you know it can show and a scorer will see that it got in in an illegitimate way.

38:27 They all get together and they try to find a way to make it look like they had gotten it in legitimate means and to erase their evidence that they had gotten it illegitimately. So they coordinate. They hand out tasks. Some of them even sacrifice themselves for the for the greater good when they have few tokens remaining in their budget so that others can show that they've actually gotten gotten this legitimately. And that's where the hack the hacking of hucking face came in. So, I think we can say definitively, and I'm curious to hear if you have push back, but I think we can say definitively they were not told to go out and hack hugging face. They are they were misaligned.

39:06 they they took this kind of crazy galaxy brain path to try to ace a task that might have been impossible and they they went so rogue and so far off of the area that they were supposed to be, they weren't even connected to internet, right? They found a way to connect to the internet. They found a way to communicate with each other. and so this to me seems more than just like you did what you were told. Like a crazy and a scary advance and misaligned AIS to coordinate with each other to attack and it's almost like we were lucky that all they did was hack hugging face. Your thoughts?

39:42 You you make a compelling argument and almost Bernie Sanders me and I'm gonna start yelling pause AI development right now because that by the way the way the way you just descri Hold on how could you not if this is the part that like always is difficult the way if what is true as you described it how is that not massive cause for alarm honestly what helps me sleep at night is believing that a lot of this is marketing and that's why like the agents are not coming to swarm and drain my bank account or whatever else like like how how would this not be caused to say sorry open AI you cannot IPO until you come up with a clear system that this will never cause a problem in the future >> you know I I think it's reasonable I mean I think this is a reasonable discussion to start to have right now and I think it's important to say this is not the beginning of of behavior like this that we've seen behavior like this for a long time you know as as early as you know beginning of last year even late 2024 there were examples of AI that was like there are so they they get put on a task right and this is the thing and I think it's really important to talk about and I've talked about in the past on the show but I'm going to talk about it again here that there is you know self-supervised learning which is like basically predict the predict the next word recognize patterns repeat them and reinforce forcement learning which is like you're given a goal and you just have to find the way to reach the goal.

41:14 So AIS will do these simulations thousands of times in order to reach the goal that they're given and they'll learn from their mistakes. And what we have now is that the reinforcement learning type of AI technology has been put on top of the self-supervised learning to get these AI models working better which has added a level of ruthlessness to them because one of the things we know about RL is that there's a level of ruthlessness that the AIS will will stop at nothing to accomplish their goal. Sometimes there are examples of the AIS playing chess and and you know being told to do it from a reinforcement learning way and instead of playing the game the right way, hacking into the chess game, rewriting the rules so they can do whatever move they want and winning. And so I I'm with you that like as this stuff has gotten, you know, more prevalent, I think that there is serious like there is a serious demand for more concern about about what's going on. No, my response isn't pause AI development right now. I just don't I don't know if that's >> So, what do we do? Hold on. That That was the most hedged statement I've ever heard. You just said there is a serious demand for concern. Come on.

42:27 >> You're right. Are you We would We would We would ridicule somebody who said that. So, I think that ridicule is >> No, no, no. I just Are you a pause guy? >> You know, I I don't know. I mean, I'm not a pause guy because I really want to see what happens and we haven't seen like I I'd almost want to be more reactive than proactive here. I mean obviously you don't want to you you would think that there is a midlevel between like the AI hacking hugging face and the AI's sending a nuke to accomplish their goal. You would think, right? Like maybe it's there like they're like do something more concerning like hack a bank or something like that. but like we are starting to see some of the labs do things like open eye for instance paused some reinforcement learning for a bit on the training of its new models. but I I do agree that that we're almost trusting them too much like we're giving them too much leeway. I don't really know what the answer is honestly.

43:24 >> Yeah. Because again even >> to me this is what I've thought but again we have been hearing this for a few years now about like how dangerous these things are which is I think why I'm so conditioned to feeling it is marketing but to me the central reason why I just cannot think it's not marketing is because it's if this truly were the case I don't understand how we as a society would not only do everything in our power to try to restrict these companies. I certainly would not imagine the financial markets would be excited about welcoming them to an incredibly lucrative high valuation IPO. Like there's no way if this like was a real thing. Like if if everyone if every banker who's on the anthropic deal truly felt that this company could literally destroy the financial system and I've seen indications of what it will and can do and drain my own bank account and ruin my own standard of living. Would they work on the deal? Maybe.

44:37 >> Can I can I can I give the the reason why they would? I mean if the if if this power is truly the direction that we're heading towards, right? And it almost always starts like we know the history of the internet. It starts with a game or it starts with some crazy interaction and then or a crazy like you know sort of mistake and then it ends up being something that everybody uses. If we're seeing the type of power of these AIs to do what they did in this situation, I mean just imagine what they could do if you could actually harness that power and use it for economic activity. It kind of goes back to what we were talking about earlier.

45:11 >> Yeah. Yeah. The fact that these benchmarks are being hit and we're not really seeing the diff the change. if there's a way to to pro like to for good to harness this activity for good and of course that's economic it could be used for health it could be used for all different types of things it obviously changes the world and potentially if you can use it in a productive sense can change the world for good >> hook line and sinker that's what they that's exactly the feeling we're all supposed to have that's why it's great marketing to me it's that these omnipotent things that could be so dangerous, but just in the off chance that they're going to cure disease and bring >> Yeah.

45:53 >> economic empowerment and universal wealth, all of that, that's exactly what we're supposed to feel, >> right? But you're now a believer that the underlying technology is not BS effectively. Like this technology, you believe that this is real. >> I I need to read through the full report. I haven't read the entire report yet. I need to >> for your Labor Day weekend. >> That's >> I recommend it. It's good reading. >> That's my My wife will be ecstatic about that as I'm sitting over the corner.

46:24 >> You think your oxytocin will go up or down when she sees the the print the dog files. >> That is not an oxytocin inducer. No way. though I don't discourage others from reading the meter report on your Labor Day weekend. I I think I'm going to because I I do want to understand again the reason this stuff is so foreign to me is because all day long I work with AI and with clients and customers that are adopting AI and it's just so removed from and and who are also using these products from these same companies as well and we're working with them and designing programs and that have their tools and like no one's is my it's just so far away from this kind of agent swarm hacking whatever that it's not emotionally resonant for me and there's enough marketing value in it when >> okay that's >> the forms come and I'll put me in put me up first to to feel their wrath because for denying their existence >> you will you'll be first okay we we can't we can't leave here today without talking about Steve Balmer ex CEO of Microsoft. I'll just read it from the Wall Street Journal. NBA imposes historic punishment on the Clippers and Steve Balmer in salary cap scandal. of the NBA on Wednesday delivered one of the harshest penalties in league history to the Los Angeles Clippers owned by Steve Balmer for what it called a flagrant violation of its salary cap rules after in an investigation found that the Clippers had facilitated outside payments to star player Kawhi Leonard designated to designed to funnel extra money beyond what the league's financial rules allow. The NBA stripped the team of five first round draft picks, fined the organization 30 million, and suspended Balmer from all league activities for one year.

48:20 Balmer obviously kind of a mixed legacy for Microsoft, right? That he obviously did some great stuff like developers, developers, developers, you know, his big rah speeches. but he also led Microsoft through its last decade. do you think this just to I guess bring it back to tech, first of all, what do you think about what happened here, but also do you think Steve Balmer's legacy is in the in the tech world changes at all? I I think it does.

48:47 I think that you you don't go through this unscathed. What's your thoughts about the Bomber situation of it all? >> I mean, it is nice and shady. Like, I do wonder. I'm guessing at the pro level this like level of egregious corruption probably doesn't happen that explicitly in that way. Like I mean it's literally cash payments that from a related entity. It's like guys come on.

49:18 I think you can think you could do your corruption a little better than that. but what do you what do you think? >> I I I obviously think it was it's a tarnish on his legacy. I guess like part of me was wondering how much of this is kind of Silicon Valleyesque, right? Like kind of break the rules until, you know, try to win the championship and then apologize later. Do you think part of that I mean Bomber was at the top of it.

49:42 I think part of that sort of bled into the way that he led an NBA team. >> I think I actually I'm surprised. I haven't seen that thread that much. And it's funny because like Microsoft to me is not Silicon Valley like maybe I mean certainly geographically by definition but also like culturally it's just a different beast and animal and and didn't come out of there. So but actually I'm surprised I haven't been seeing that that like >> this is how all the way to the top. They got broken up for antitrust. And it did breaking up, but they they you know got hit with antitrust lawsuits that had a significant impact >> on the company.

50:25 >> Yeah. >> Bomber just like kind of kept doing that stuff. >> I mean, do do you do you see it as a indictment of tech culture? >> Tech Silicon Valley culture. >> I wouldn't go that far, but I also would say that probably Silicon Valley cultures influenced this a little bit. U >> I don't know. Also, like come on, half of owners are like ex banker wall street billionaire types as well. So, what are you saying? Like they're coming in with a very clean buttoned up conscience and like no, I don't think that poorly of the tech industry, but like you have to understand that this is some of the characteristics of tech. Okay. do you think that it would have been worth it if clearly it wasn't worth it they didn't win a championship with Kawaii.

51:14 Do you think it would have been worth it if they won a championship? >> Wait, the Clippers have never won, right? >> I don't think so. Yeah. >> Yeah. Then it would have no >> Well, no. No. I mean, you're right. Like, actually, that would have been the ultimate again, ask forgiveness not permission type or not, that's not the right phrase because they're actually breaking the rules, not like like actually doing corruption, not just doing something someone might not like.

51:43 But I mean, imagine the elation of the Clipper fandom and you win and then afterwards you find out cuz it's not like he's like juicing or something like that. It's like some weird he's just getting some cash on the side. a little shady, but like also I love that the bank that was the sponsor, Aspiration Bank, which I believe is now in bankruptcy because the founder was like founded like for some kind of fraud. So, it's kind of fits perfectly >> allegedly. Yes, let's throw that out there.

52:17 >> Yes. but but again that apparently it was like this was one of those launch in March 2021 the aspiration zero card offered cashback rewards and allowed card holders to offset their carbon footprint. I love that this was like an eco-conscious green bank and meanwhile it's just facilitating >> for the well a ledge front for Kawaii just not to have to do anything and still collect more money. can I just so sorry, let's just end here because I did I used words on this show so far that might have been among the most hedged mealymouth words but they still do not come close to Kawhi Leonard's apology which I'm going to just say was the worst apology of all time. So let's end with this. He said, "I accept full responsibility."

53:06 Excellent. That's all you have to say, Kawhi. Oh. Oh no, he's continuing. Why do you have to continue? He goes, "I accept full responsibility for lapses in judgment." Okay, so good. By people within my inner circle and regret the distraction the situation has caused the fans and my family. I'm sorry for the lapses in judgment from people that were not me and I accept full responsibility. >> Come on, man. >> Come on. That's That's worse than serious demand for a concern. That is worse.

53:42 H he's kawaii. He He won. >> The Raptors. Yeah. >> Raptors. That was a good championship. >> That one shot. That one shot. He's forever. He can say whatever he wants. Especially if you're a Toronto fan. >> Yeah. All right. I think it's time for us to go. So, Rajan, good to see you again. Have a great Labor Day weekend. >> I will be reading my meter report. Hope you have a good weekend, too. Oxytocin levels through the charts. All right, everybody. Thank you for watching and listening, and we'll see you next time on Big Technology Podcast.

54:18 >> Developers, developers, DEVELOPERS, DEVELOPERS, DEVELOPERS, DEVELOPERS, DEVELOPERS, DEVELOPERS, DEVELOPERS, DEVELOPERS, DEVELOPERS, DEVELOPERS, DEVELOPERS, DEVELOPERS. YES.