Section Insights
The Urgency of AI Development
Why is there a need to slow down AI development?
The rapid progress of AI models raises concerns about their potential dangers. Experts emphasize the need to slow down development to ensure safety and ethical considerations.
- AI models are advancing at an alarming rate.
- There is a consensus on the need for caution in AI development.
- The implications of AI could be profound and require careful consideration.
Recent Advances in Mathematical Problem Solving
What recent advancements have been made in AI's ability to solve mathematical problems?
OpenAI has recently solved 722 out of 4,000 hard math problems, showcasing significant progress in AI's mathematical capabilities.
- AI has solved landmark mathematical theorems that have stood since the 1800s.
- The rapid solving of complex problems indicates a shift in mathematical research dynamics.
- Mathematicians are struggling to keep up with the pace of AI advancements.
The Role of Policy in AI Development
What is the importance of policy in the context of AI advancements?
Policy responses are crucial to manage the risks associated with rapid AI development and ensure that ethical considerations are addressed.
- Policy must adapt to the fast-paced changes in AI technology.
- There is a need for proactive measures to mitigate potential risks.
- Engagement with experts is necessary to shape effective AI policies.
The Future of AI and Public Good
How can AI be aligned with public good?
AI development should focus on public good rather than solely profit, emphasizing the importance of ethical considerations and societal impact.
- AI should serve as a public good, addressing societal challenges.
- There are opportunities for collaboration between AI developers and public interest organizations.
- Ethical AI development requires transparency and accountability.
Agent Identity and Cybersecurity
What are the implications of agent identity in AI and cybersecurity?
Understanding agent identity is crucial for cybersecurity, especially in the context of AI systems interacting with each other.
- Agent identity verification is essential for trust in AI systems.
- Cybersecurity measures must evolve to address AI-specific challenges.
- Collaboration between AI and cybersecurity experts is necessary for effective solutions.
Data Utilization in AI Development
How is data being utilized to improve AI models?
Data is increasingly being used to benchmark and enhance AI model performance, with a focus on economic value in various domains.
- Data quality is critical for improving AI models.
- Benchmarking helps assess AI performance in real-world applications.
- Innovative data production methods are being explored to enhance AI capabilities.
Training Methods in AI Development
What are the current trends in AI training methods?
There is a shift towards using reinforcement learning (RL) and supervised fine-tuning (SFT) to enhance model training while minimizing disruption.
- Different training methods can impact model performance and stability.
- Enterprises are exploring various customization strategies for AI models.
- Balancing SFT and RL is key to effective AI training.
Human Contribution to AI Performance
In what domains has AI surpassed human performance?
AI has reached superhuman performance in domains like chess and Go, indicating a shift in reliance from human expertise to machine capabilities.
- AI is increasingly outperforming humans in specific tasks.
- The collaboration between humans and AI can yield superior results.
- Understanding the limits of human contribution is essential for future AI development.
Economic Implications of AI Pricing
What are the economic implications of AI service pricing?
The pricing strategies for AI services reflect their perceived value and potential for widespread adoption across industries.
- AI services are becoming more accessible, leading to increased adoption.
- Pricing models may influence how organizations integrate AI into their operations.
- Understanding the economic landscape of AI is crucial for strategic planning.
Transcript
0:04 It's pretty crazy. >> It's just insane. It may fake my books. >> Not AI. It's >> slop. The models, they just want to learn. The models, they just want to learn. They'll do it. They'll do it. The models, they just want to learn. the models. They just want to learn. They'll do it. They'll do it.
0:34 >> This is the most dangerous thing that humans have ever gone near. We have to slow down. >> You're not talking to somebody who woke up a loser. Woke up a loser. Woke up a loser. Woke up a loser.
1:06 Woke up a loser. Woke up a loser. The models, they just want to lure them. The models, they just want to lure them. They'll do it. They'll do it. The models, they just want to lure them. The models, they just want to lure them. >> They'll do it. They'll do it. >> We need to slow this down. >> We have to slow down.
1:37 Researchers are continually being surprised at the rate of progress. That's sometimes called the hard takeoff scenario where maybe they're extremely fast at doing that. >> Things are moving really fast. >> So, they're immortal. We've actually solved the problem of immortality, but it's only for digital things. >> Succession to digital intelligence or augmented humans is ineffable.
2:07 >> And the day will come when AI will do all of our all the things. Not just some of them, all of them. We have to slow down. We have to slow down. We have to slow down. We have to slow down. We have to slow down. We have to slow down. Slow down. Slow down. Slow down. Slow down. It's insane the pace and there's no reason to be this fast. >> The models, they just want to learn.
2:32 >> We have to slow down. >> The models, they just want to learn. Let's just go for it. And good morning. It is Wednesday, October 7th, 9:02 a.m. Nathan, good morning to you. >> Good morning, Pash. what a jam this morning. I was impressed. >> it is I was using up my tokens. I had some tokens left over from Fable. I gave it to I I I gave it to Fable and it picked maybe like seven or eight clips from like the last like couple of years. you know just randomly like stuff that stuff that I remember like the key the key moments right like Jensen's like you know I didn't wake up a loser etc etc. and I handed it off to Fable and that's all that I did. and I I I had I I told it to I told it I was thinking about there's a song from the 1970s from Jill Scott Harren called The Revolution is not televised. It's a funk song. and I said, "Well, I'm thinking about this, but I want you to do something new, obviously, and you just figure it out."
3:40 So, it figured out the hook. It figured out which places to cut. They're like two minute segments of you know, vocals. It figured out which which were the interesting parts to cut. it transcribed into deepgram first. It you know cut it and then it put together one version and basically I probably listened to about nine versions of it. So each time I listen to it I be like I don't like this, don't like that and finally got it to this date and yeah that's and what it's doing is it's not doing the puno thing. It's not it's not doing generative AI. I gave it access to a API API based digital audio workstation which is what you use to kind of compose various instruments together onto a track. And I told it to use the digital audio workstation. It's called Reaper.
4:30 It's open. It's not open source. It's like a shareware. I said go ahead use this. You can play the instruments. Figure out which instruments to play. you if you need some samples here are you know CC0 sites where you can download samples in order to use and go compose something. So it actually compose something with instruments rather than and it's and it can't hear obviously. So it's it's basically using the you know just as music theory and kind of the the waves I guess the the par transforms and you know what it can see from you know the frequency diagrams and it put it together that's that's that's the solve. So >> it's pretty amazing. So the the software that you were using is desktop software or it runs in the cloud like is it interacting with computer use? You said API or how doing that?
5:25 >> It's a it's a desktop software called Reaper. it runs on Windows, it runs on Mac, it runs on Linux and it's just something that you can use. You can interact with it via Python, which means basically there's underlying the GUI there's a command line interface there that you can use. and that allows the the AI agents to use it you know almost almost a full all the full you know functionality can be used via the command line. So and that is really how the agents are interacting with with the software.
6:02 >> Wow, pretty impressive. not to put you on the spot, I don't know if you have the the different versions handy, but do you have like the first version? I'd be really interested to hear the first version, maybe like a middle version a little bit and then the final version and >> I think so. let me let me >> just compare and contrast would be pretty interesting. >> I have one draft here. Let me let me just give me one moment and let me put it up. Just give me a second.
6:31 So it is interesting that we've been we've been talking about this for a while now. and we've been doing suno for a while and I think sunno is real generative AI. This is not this is not really generative AI. This is using, you know, it's it's using software, right? It's not it's not really generative AI, but perhaps the fidelity is maybe a little bit little bit better. I don't know. than simple than than than generative AI because it's it's really like playing the instruments that and that's what that's what really surprises me. It's it's actually composing multiple instruments together.
7:11 All right, let's see if we can play this. >> The models, they just want to learn >> little jazz flute. >> Open AI started. I met Ilia. One of the first things he said to me was, "Look, the models, they just want to learn." >> So, this was the first one. >> Will never grow up in a world where they were smarter than computers. >> We will have for the first time something that is smarter than the smartest human.
7:36 >> Pause this for a moment. And I think the main the main thing from the first version was there's a lot of like little glitches here and there. and also for example, there's a Terrence Tao saying it's just going too fast. and what I wanted it to do was after each it's going too fast, I wanted it to speed up the tempo, which was kind of a inside joke, right? So, and and that was something that it didn't quite catch as in like, you know, it should be it should be doing something with the tempo when it says it's going too fast. So that was that was one one change that I made. And there was a lot of like small little clips where it had included the first like quarter second of the next word and so the sample was like a little bit like misshapen. And so that required me telling it like look you you accidentally included a little bit of the sample so can you figure out what the two voices are and you know disagregate them. it's it's pretty good. It it has it has a you know Saturn u interview in there where he actually has underlying music playing during the during the clip and it actually took out took out the underlying music too.
8:49 >> it's I mean it's all it's all a human could do this, right? Like let's be let's be honest. A human could sit down and but I couldn't do it like >> Well, not this human. Yeah. No, >> not this human. >> Some human out there might be able to do it. >> Yeah. Some some human out there would be able to do it, but I I would not. so yeah, it it is the the the instrument playing has really gotten to me because it indicates to me that we're going to see non-generative AI music, which is AI actually playing the instruments, like legitimately playing the instruments.
9:22 which is probably may may not sound like generative AI. Maybe the generative AI has like a little bit of a you can detect or whatever. And maybe maybe we'll get like actual instruments playing and actual composition. which would really be quite quite different from you know the next lay the next level of sophistication basically the actual playing of instruments. >> It's pretty impressive. You want to play a little bit more of the final version? I just want to hear a little bit more.
9:49 >> Right. let me the let me have the this one. >> I'll I'll skip it. what would be a a slight lead. >> Oh, now we're fast. You know, a few days acceler could suddenly become a chasm. >> The models, they just want to learn. >> Hard takeoff. >> Some people say it's in three. Some people say it's in five, 10. Numbers have been thrown around. >> You talk about 2030, like I don't know what the world looks like in 2030. Hard take off.
10:22 >> We have a brain. The brain is a biological computer. Should we consider them humanity or different from humanity? The >> models they just want a world. >> There's not going to be a clear distinction between human and machine. The bottom line is we are one human machine civilization. >> Is that Curtzswe? >> This technology has already expanded who we are and it's going to go into high gear when we get to the steep part of the exponential >> and end.
10:52 >> Wow, that's pretty cool. I mean, I love Sununo. I was just saying the other day I think you know of course the core frontier models are you know core to my life at this point but the AI app that I could live without practically but which I go to all the time and would have to say is my favorite AI app at this point iso because I just can make things there that delight me and stick with me and that I even like now sometimes put on when I go for a run. and I just could never do that before and I I so I love that. But this is like pretty impressive as well. And I'm sure there's a lot of synergy there to be explored too. I mean, I've only ever done one sample with Sunno where Claude tipped me off to this. It was for one of the China episodes. the idea that I had just super high level was like, can we sample some sort of classic Chinese song that everybody would know, but make it into like a Kanye sample, you know, based on that historical artifact. And it was able to do that. That actually came out pretty well. So, a lot of these clips you could also throw in there. And then you can do like, remixes and covers.
12:09 So there's a lot of opportunity, I think, to use something like this as a base to really kind of dial in more precisely from the beginning certain structural elements that you might want and then still, you know, have the generative AI on top. I assume that what this software probably can't do at this point is vocal tracks or correct me if I'm wrong. >> Yeah. Yeah, it can't it can't do vocal tracks. It can't compose vocal tracks.
12:32 I I think I think it's sunno I think the suno songs came out much much better. I think I think the main thing about this is that I chose samples which were relevant and like and like we know we know a lot of these voices. We've heard a lot of these clips before >> and so and so it's much more topical for for us and for people in in the industry. but I think the the sophistication of Sunno and the the level of like bangeress inso is higher than than what you could compose than what the what what the machine was composing because the machine did all the composition like it wasn't wasn't me. and so the machine was the one that came up with the sequencing. It chose which samples to use. you know, it had like minute long from each minute, two minute long samples and it shows the specific phrases. I told it to put together a narrative. I wanted like a kind of story line. so it starts off with Dario's like the models they would just want to learn and then it does his that's became the refrain and then it it did the the intro part where he said how Ilia told him about the models that they just want to learn and it ends with Kershw which is a very kind of you know sober like you know we're progressing towards a human machine you know civilization. So, it put together the storyline and that I think was pretty interesting because I said, you know, this is our show.
13:59 you know, put it put this together. These are the samples. I kind of want a narrative from beginning to end. And then it came out came up with that and then a few changes here and there. So, my my bit was probably, you know, Terrence Tower saying we got to slow down and each time he says we got to slow down, it speeds up. So, that was my that was my contribution. net contribution. >> It's pretty impressive. I think this is I mean obviously it's a curiosity and on one level you know who cares. It would be easy to say you know this isn't moving GDP. This is just like us you know arguably amusing ourselves to death in in a new form. So I do think that perspective is like somewhat valid. But also, I do think this goes right to the heart of one of the big questions that everybody's been talking about recently, which is simply how good are these things going to get in domains that are not automatically or readily verifiable.
14:58 Obviously, you know, I don't need to tell you that we've seen at this point just an overwhelming amount of math results in not just recent weeks, but last 24 hours. But there's been this concern or not necess I mean you know in some ways it might be might be good for the for some for the future in some ways if some of these other things prove to be harder but it really does seem like the generalization is pretty strong. I mean obviously we don't know what the mix is of data that they're putting into the frontier runs and I'm sure there is some music in there. Joel Borggan we talked to you last week said that prior to the latest generation of models they could never transcribe sheet music and now they're doing that pretty well also. So it seems like there has been some sort of dataation that's made a difference but I highly doubt they're doing much in the way of feedback loop on music. This is probably one of those things that they just kind of threw some stuff in, see what comes out. Okay, it's better than last time.
16:12 Great. Let's ship it. pretty impressive and definitely strongly suggests to me that we are not going to be able to hide behind the, you know, well, it only works for verifiable reward type tasks for really any longer. I mean, if you were thinking if you're thinking that was going to be the the last wall to hold, I I think unfortunately it's not going to listen to the listen to the track once more if you don't believe me.
16:46 >> so one one fascinating thing. So let's let's get to the math results. we we are we have gotten a bunch of math results in the last 24 hours. So basically what has happened is well before we move to math results >> music and math are often very connected actually. So a lot of a lot of you know music theory is is math theory and so I wonder if actually improving dramatically on on on on math is also going to have impacts on music you know almost automatically. but let's just let's just move on to the the math stuff. In the last 24 hours, what has happened is OpenAI has done a the first drop of perhaps many and they have dropped about 722 theorems. so they have a a set of hard math problems about 4,000 of them and they've solved roughly about what 20 20ish% 722 of those problems. they were solved by a model running chat GBT Pro for no more than three hours on average about median time about three hours each.
18:00 it sounds that sounds as though like it's a pre-release model by the way. So the fact that it has a pro and it's not just you know internal model number one or whatever it sounds as though it is a it is a pre-release model and the model solved about 722 of them. It the number of problems solved is so large that mathematicians are having no no single mathematician is in enough of these fields to give actual context anymore.
18:33 some of the results I think probably about five or six of them are landmark results for things which have stood since the 1800s. and you know Leavant Alpagi at at anthropic a mathematician he said that you know this is about the same scale as the discovery of the zero or the invention of like proofs like the like a method to do proofs or the invention of the computer and applying it to math about the same as you level of dramatic improvement in the last 2,000 years as one of those, you know, landmark events.
19:23 And people are still having trouble digesting it. Some some are very basic things. So for example, how do you multiply two integers together? Like what is the fastest way to multiply two integers together? for the longest time, there was a certain limit and now that limit has been exceeded. That limit has been exceeded. But by very small amount, right? By very very small amount. If you did, if you were to multiply two inteious and the integers was so large and across that entire like very large larger than like the size of the universe kind of numbers, you would have like a small smidgen of improvement. But the the key thing is that the the barrier has been broken.
20:04 And after the barrier has been broken, people are immediately getting better results. So already in the last like 14 15 hours people have improved on those numbers using using the techniques there because because the techniques were not fully milked like as soon as they broke the you know broke the barrier solved solved it was published but the techniques can still be built the the techniques that had been revealed can still be milked for more improvement and that's what's happening right now. So humans are optimizing as we speak.
20:33 very dramatic I think. and it should be front page news on every newspaper. Some mathematicians believe. it is not. It is nowhere on the front. >> I mean talk about a communications challenge, right? It's like God help you. yeah, I mean we will we we will see what the impacts are but we are starting to see some people are starting to see starting to say that we are about to see very dramatic things like you know within 5 years we'll be able to prove whether or not faster than light travel is possible or impossible.
21:11 you know we might solve the unified field theory or in gravitational and gravity by next year. So all these things in theoretical physics and you know science might fall. It also suggests to me that there was a lot of low lowhanging fruit actually fair you know fair fair amount of low hanging fruit in in mathematics. It suggests to me that this is going to be true of every other scientific endeavor too. A lot of these solutions were using techniques from different areas and combi and recombining them in order to solve problems. I suspect that it's going to be true of I I I think some people are like oh bio is not going to be like this because you know you still need experimental data. I believe there should be a lot of lowhanging fruit in bio as well because a lot of bio people were not very mathematically inclined and so a lot of mathematics has not been applied to a lot of bio and I believe there is enough corpus of experimental data for you to start applying some of that. and so I suspect there will be a fair amount of discoveries in bio too. you know even before requiring you know deep new experimentation. So >> yeah wild times when I talked to the founders of harmonic which is one of the super intelligence for math or ma you know mathematical supermut >> oh really I'm not muted.
22:36 I'll >> mute and unmute. >> How about now? Can you hear me now? Hello. Hello. Want me to refresh and come back? >> I can hear you. >> Well, >> I'll refresh. >> I'll come right back. >> All right, I'm back. Any better? >> Yeah. >> Okay, >> I can hear you, I think.
23:08 >> Cool. So I was just saying when I talked to the founders of harmonic, a mathematical startup, their vision for the future was that we would have a super abundance of theory where we would get to the point where we'd have multiple competing theories of everything >> would then only be resolvable by physical experiment. But their their hypothesis is that there will be multiple self-coherent theories of everything that AIS will generate as candidates consistent with the evidence that we have and then we'll need more evidence to break the these you know competing galaxy brain frontier theories which you know I mean it sounds pretty far out but this is I think pretty much what you would expect to see if you were on that path. though certainly it seems not too insane right now. One other thing I wanted to just pull up real quick is a pretty striking chart. I mean in terms of things going vertical.
24:17 do we see the Do we see the share? I don't see the share. Do you see the share? >> No. No, not yet. >> It says I am sharing so I'm not sure. Oh, there we go. Okay, >> there we go. so this goes back to our classic debate that you know we've touched on several times like if you are at least maybe a variant on it, right? We've previously covered like if you're a doomer what are your trades says Tyler Cowen. my counter to that previously was having frontier models go out and try to figure out was there any way to trade through World War II Germany and Japan and come out the other side having won. And the answer was basically no. Like you you can't really do that when things go that bad. You know, the markets get closed, everything gets kind of confiscated. you know, nobody pays out basically. There's another kind of variant on that question which is like where is AI in the macro statistics today and obviously there hasn't been too much of that so far but the debate continues between Tyler and Dean apparently Dean was at Mercada Center before he went to OpenAI and look at the movement in the US 30-year bond over just the last year. I mean, this this has not moved that much in a long time.
25:53 And it for my entire life pretty much, it's been a secular downward trend with, you know, of course little blips, but like >> basically when I was born, we were in sort of high teen percentages and, you know, Vulkar and everything else and just brought it down down down down down all the way to the zero interest rate period a couple years ago. And now we are all the way back up to approaching 6% and like you know it's in this month alone it's jumped by a full half a percent and I don't know when the last time we've seen a jump like that is. So this is definitely something to watch and it does you know it suggests that like obviously the the scale of the data center buildout is getting so big that it's like pretty hard for it not to have an effect on just overall global capital measures and the ROI on these data centers is so high right now.
26:54 I was listening to I was a little late to hear this one, but the two Patels Darkesh and Dylan had a podcast not too long ago where they talked about this and the basic upshot was like the pressure that the extreme returns to data centers are generating right now is going to crowd out a ton of other stuff. because you can right now with a new data center given the prices that they're paying and the margins that they have. The frontier companies can basically make their money back on a new data center buildout in roughly a year and then they've got you know four plus years of useful life after that where they're just making unbelievable bank.
27:41 And so, you know, they they talked also, as you have commented several times, that like the underwriter or, you know, the whoever's balance sheet is really financing this stuff kind of gets to dictate the terms, but the return is just so insane that money continues to flow into it and everybody else has to figure out how they can compete with it. And we're seeing that for US treasuries to compete with it, the rate has to go up quite a bit.
28:11 This is going to create borrowing problems for all kinds of other entities. I mean the US government might even have some problems like the interest you know when interest goes up this much like it makes a huge difference on the the debt burden you know to service the interest alone is like a is a problem and it's becoming a bigger problem like by the month at this point. But now do that same analysis for other countries that don't have the luxury of you just monetizing their debt or for companies that are trying to make investments.
28:43 Everybody's going to be paying significantly higher rates. And this does also give some credence I think in my mind to the worry about the permanent underclass. which I still think is like very seemly very unsemly for people to be like prioritizing their own you know personal position relative to the the you know over or underclass of the future. But you can start to see how it shapes up here. I mean, the old pickety idea that people who have money make more money and that, you know, returns to capital are just going to continue have gone up and will continue to go up. and you just get bigger and bigger pools of money that way. You can kind of see that here, right? I mean, if you're sitting on a lot of cash, all of a sudden you're getting like close to 6% in what is in theory risk-free return. Obviously, there's no real risk-free return, but this is as close as it gets and the premium is moving that much. I find that to be a pretty shocking indicator.
29:50 >> So, a couple of thoughts. One is, what happens, right? Like, let's say it goes to 10%. Right? Let's say it's continues climbing to 10%. At that point, you know, will policy makers step in somehow? will there be cross subsidies? Will there be taxes on data centers to transfer some of that wealth?
30:21 something has something's got to give, right? You can't the there are other alternatives. being a distressed former distressed credit trader what ends up happening is bankruptcies and especially in the United States bankruptcies are it's not the end of the world it's actually you take a stay on paying anything and you restructure to something that works for your level of profitability. So let's say if you if you were servicing $3 billion of debt with $100 million, you you know write that down to like a billion dollars and then you're fine, right? So it's a you do capital restructuring. The people who hold the capital take losses. So and so if you were a bond investor and you had you know you were now able to invest in these bonds that are giving you 10%. On the other hand, your bonds that you b you bought like last year which are giving you like 2% someone you know when they try to refinance they default and then you lose some of the capital there. So you kind of you know balance off the portfolio between two sides. companies change hands. So if you are if you own the equity in a company you start to you know you start to lose it. so a number of things start to happen I think. anyway I I I think we have our our first guest. so let's let's get him on.
31:44 >> Yeah. While you do that, one one final thought on this topic was just and I don't think we talked about this yesterday, but a takeaway from the curve this weekend. Also, one of the, you know, not main threads that I was involved in discussing, but a general vibe was there is going to be government intervention. and you know, obviously we've already seen some, but the idea that like Congress will never act, I think was that used to be sort of taken for granted. And now the new vibe that I was hearing was like after this election, especially if the Democrats take both houses, both both chambers.
32:26 that may change and you might even see a bunch of Republicans join the Democrats because really in their heart of hearts they want to do something. They don't really want to go against the president and there hasn't been a lot of people willing to take that chance yet and they're also getting a ton of money from the data center interests right now. but that all could come up for a renegotiation and realignment after the midterm. And we might actually see congressional action a lot sooner than u maybe a lot of people who've kind of grown accustomed to the idea that it can never happen or it'll be, you know, stasis forever might think. So, I do think policy response is definitely part of what we need to be looking out for. But with that, let's bring Evan up.
33:13 >> Indeed. Okay. And we got to get that track on our motion graphics. By the way, when we do the motion graphics package, I want to hear the I want to hear the models just want to learn. >> Indeed. let me let me introduce Evan Miazono. Evan Miazono is the founder and CEO of Atlas Ignoda, the successor to Atlas Computing. Atlas is a nonprofit that looks for consequential AI risks without a clear institutional owner, develops a possible intervention, and recruits someone to carry it forward. It operates within convergent research, which helps launch organizations built around specific scientific and engineering problems. Atlas's early work focused on using AI to help produce software with mathematically checkable guarantees. The team recruited and supported Jason Gross in experiments that helped lead to theorem labs which works on AI assisted software verification. Atlas also helped initiate OATH technologies led by Mike Dods to build tools for formal specifications and led by MT Sakhan to develop hardware that responds to physical tampering.
34:33 Before Atlas, Evan built research and meta-cience programs at protocol labs whose work includes networking, distributed systems, and cryptography. He also created its network goods venture studio which helped launch projects including hyperserts and funding the commons. Earlier he completed a Caltech PhD working on optical quantum memory. His undergraduate years included material science, mathematics, philosophy and set design and construction. Across his recent work, a recurring concern is how useful technical ideas acquire the people, funding, and institutional support needed to become something people can actually use.
35:21 >> Evan, welcome to the show. >> Thank you for having me. Excited to be here. >> >> so much ground to cover. Yeah, >> you've been prolific. I I feel like I should be far better at sticking to a specific narrow topic and there are some threads that I think are reasonable through lines around pricing supporting and accelerating public goods and it just kind of there's a version that where what those public goods are has simply shifted but yeah it felt it feels like I end up talking to experts in a new field roughly every like two to five years.
36:04 >> I I mean do doing this show, we're talking to experts in new fields every half an hour and I feel like my worldview is updating, you know, faster than I can sometimes consolidate the learnings. >> yeah, that's the challenge of the run-up to the singularity. Let's maybe start with, you know, my wife has been on this recently. She she recently attended an event and her takeaway coming home was it is really time for people to rethink what they're doing. A lot of momentum, you know, a lot of people have, you know, got projects going and as we get close to, you know, potentially a a crunchtime type atmosphere, it might be time for a lot of people to reallocate their time and attention. So, I'm, you know, think trying to bring that challenge to myself to some degree. But you've done that recently, right? you have kind of taken a step back, restructured your organization, relaunched. tell us about that process, you know, what caused it.
37:01 it's obviously hard to do. You went ahead and did it anyway. I'm interested in that story. >> Yeah, I think so. I had set out partly due to close relationship and advi mentorship and collaboration with David Dal Rimple u aka Davidad who as he was leaving protocol labs to go start the safeguard AI program at Arya. he and I agreed that that was a good place for that program and that there should be a nonprofit to partner with it to try and accelerate the relevant aspects of foral methods and AI to make it easier to use various foral methods. Also the the text on the bottom I was technically not going to to safeguard an AI. I was like I I I went to start Atlas to go do that. The plan for Atlas computing was to prototype a bunch of things and make it much easier to start from like where the world was now and get to the world that David was trying to start assembling pieces for.
38:15 The challenge became that while I was trying to build prototypes and things like that, people kept saying, "Hey, there's a like there's a clear need for someone to do X. No one is doing it." And I say, "Well, I can go find someone to do that. Like, let me make sure that this is the right thing to do first, but then like get maybe get them the experts they need, the resources they need." And I did this three times before someone pointed out to me actually the I had it brought in a chief of staff who helped me transition from Atlas computing to Atlas Ignod and he said what is your unfair advantage?
38:56 Why aren't you just doing more of that? And the that line of questioning led me to realize that most of the there are many aspects of the world in which people will say strategically like I want the world to look like this and then completely separately they'll look at what they could do and then pick the obvious thing in front of them rather than trying to backsolve from where the world they're trying to get to. And I've recently been telling a lot of people we I think a lot of people are tactical when they think they're being strategic and realizing that there is actually this big failure mode for a lot of the philanthropic money that's moving into the AI resilient space generally where if you put up an RFP everyone will take the problem that they care very deeply about their passion project and then wrap it in the language of their solution of your of the problem you post and then submit that. Same for requests for startups, same for government bas.
39:59 And the question becomes, who is out there actually trying to define here are all of the possible options? Here are the best ways to achieve that and then who do I hand those off to? I think that a lot of the mechanisms don't do the mechanisms we have around markets don't do a good job of allocating resources in a way where there's no value capture. And I think that many nonprofits that try to or don't have the proper incentive to reduce the amount of claim they have on spinouts and things like this.
40:39 how do you select the important problems? Like what is the what is the framework that you use to select the problems that you feel deserve your attention and that you want to recruit people for? >> There are two major factors. I think the the most obvious and the clearest one is that as a nonprofit we're pretty much under at the will operating the will of the donors. If a donor comes to us and says we care very very deeply about what h what happens with cyber security if we kick off recursive self-improvement in six months then I can go and look at what seems most catastrophic. There are some very scary scenarios where the electrical grid turns off and maybe doesn't turn back on in most of California. And so that seems like something that deserves some attention. The question there are many many people who are thinking about that problem and related problems. Many of those people are don't necessarily have the context and resources to fix all of the problems they come across. And so having an organization out there, ideally there would be multiple organizations where someone can bring oh hey like I know about this problem but I'm not in a position to solve it or oh you mentioned this problem I actually have a solution that seems viable but I can't pursue it or someone saying I have resources and I care very deeply about this problem. This is I I've been claiming repeatedly that as intelligence gets cheap, it's the coordination that gets expensive. And so having shelling points around who does this coordination, becoming one of those shelling points or creating one seems very very useful. And so there's some amount where if the the metric for should I work on this is do is there a funer who's going to prospectively or retrospectively fund projects and efforts on this idea where the retrospective is like if I produce a team that can execute but they need $10 million if I can figure out a priority that no one is going to spend that money then that isn't a project worthwhile but I There are many aspects where there are resources but they just you need people who can find what those resource where those resources are and know how to invoke them. Same for talent same for expertise and awareness of the problems which aren't likely to exist in the same people. One of the things I've noted is AI resilience, AI safety is a very broad topic and I think perhaps maybe the funders are interested in the topic broadly but they don't know how to narrow down to what are the specific verticals that you need to work on. So, how do you how do you match those two like this general kind of I want to help on AI resilience or AI safety versus like these specific vertical projects that you have to like okay you know this is worthy of funding and this is achievable within a two-year mark or you know because it doesn't make sense to fund something which is going to take 10 years perhaps right so how do how do you narrow down what you what to work on >> yeah I think I have certain certain assumptions around what the current state of play is, what the first and second derivatives on AI capabilities look like, and I definitely screen for those things when I'm thinking about scaling my team. I I think that most of the funders that I've been talking to share fairly highly overlapping assumptions.
44:46 it's very like the funders themselves obviously don't have overlapping assumptions with each other. but I think that there are And I would say I've also mostly been speaking with our funding comes from coefficient giving openai foundation survival and flourishing fund and the ex effective institutions project and so they all have more than probably average like more than a traditional nonprofit's amount of context on what they care about what they want the world to look like. For someone who like if I entered into a conversation with a relevant person at the Gates Foundation for instance, I would be talking through here are the things you have expressed interest in on longer time scales. maybe that's economic freedom and access to a higher standard of living. Maybe there are things about preventing miss and disinformation or improving cognitive security. Maybe there's something around concentration of power or new interests like loss of control. And then identify these are things that I think that we've come across that no one is working on.
46:02 These are things that would very much benefit from support. I need either money to hire people to go and chase down these ideas. I I found that if you say here is a problem and you go talk to experts and you say what can be done about this you will get a lot of ideas you will also get a lot of bad ideas and it is very easy for experts to oversimplify subsets of civilization that they don't have deep familiarity with and make assumptions about what is adoptable or what isn't in ways that I think are very important to whether or not an intervention is useful, feasible, etc.
46:46 And so having people who can very quickly use their model of how big organizations make decisions, how technology actually gets adopted and things like this. And being able to quickly test out these ideas is very useful. And then on top of that, getting from okay, this seems very this seems plausible. Can we get conditional commitments? If we build this out, will it actually get adopted and used and have the impact we intend? and I think that this works across a lot of different problems. Sorry for the meandering answer, but hopefully give some helpful context.
47:25 >> Let's talk about just some of the big gaps. You know, what are the things that right now you think are highest on this? I guess maybe you score him by sort of a product of importance and neglectedness is kind of the the takeaway that I am getting. >> Also the the timeliness I think that there's like a question of will if we do is there something that can be done in time. and I would say one of the things that I'm probably the two things that I've been spending the most time on personally recently are what can or should be done around protecting critical infrastructure especially from accidental agent swarms and also things malicious use of openweight models. I have raised an I think there's an interesting question of what happens if instead of open AAI accidentally attacking hugging face we have the companies behind Deepse models accidentally attacking state department servers.
48:33 How do you know if that's an accident? What do you do in response? What would you do if or what would we do if we saw some malicious attacks to critical infrastructure coming from an open AAI server? How would you know that it was this and not coming from somewhere else? How would you shut it down quickly? What if like with varying levels of escalation? I think that there are many it's very easy to come up with ideas for things that one could do and much harder to find people who can say like yes that is absolutely feasible or it isn't. I can go into more of those but I think that is like a cluster of scenarios is very interesting. and one of the interventions you might add is could you use DNS to have cryptographic attestation of whoever is providing inference. I think this would be very useful and also a great complement to having DNS for people. So if your agent is talking to my agent, they can verify that this agent in fact is my agent. Like your agent can verify this is EMO's agent being signed off on by some key. I think that there are some shortcuts to adoption that you can take if you're not trying to build this into a startup that returns the fund.
49:48 >> And so this looks much more like a public good. I think that un different entire set of of interests well I guess kind of kind of blurred in the two things I wanted to highlight were one things around agent identity and then also I think some of these responses to short timeline cyber combination of awareness combin like some amount of I think it would be really good to have the like secure the perimeter secure the area and then do forensics on a like multi- aent swarm event. I know that I expect me will be doing a lot of slot investigations and it would be very interesting to make those higher confidence both on the like what can be done in advance of one of those events. Can we make it very easy? I also have spent a lot of time thinking about formal verification and all its various uses and applications and I think that there are some potentially exciting directions where the kinds of things that you would want around to the extent that I agree that government is very likely to get involved in a lot of these things. We might want a certain set of models that only can be run if the companies that are running them are are can provide strong evidence that they are applying a one of a certain set of evals or classifiers on those.
51:28 >> Evan >> and the question become I can hear you sorry. so it strikes me that for agent identity it strikes me that during the hugging face attacks the agents actually came up with they started suspecting that some some people were basically poisoned what they call poisoned and they wanted to have some form of cryptographic identity I I believe they they they started to have some kind of signing keys etc. They didn't fully flesh out a identity system. it strikes me that you know they kind of emergently came up with this idea and perhaps that for larger swarms it is impossible to operate without some form of identity verification even within within the swarm perhaps. it also strikes me that a lot of the there's a lot of like tensions between privacy and identity right in the to the extent that for some things you want not to have that identity and we've had this kind of back and forth about whether you should have KYC to purchase data center capacity for example the Biden administration had an executive order they wanted to do proceed with KYC and then that got you know Trump administration came in they cancelled that it sounds as though the cryptographic identity of agents is kind of a follow on to KYC for data centers would you would you say that is >> I think that there definitely feels like there's overlap when I have conceived of KYC I will note that important to note that I am in conversations ranging from how what do stricter monetary controls and financial controls on agents look like so that we can prevent agents running like creating and running a shadow economy on the one hand and also risks on the other hand of how do we have identity that would be useful for things like this but still can also support refugees and and dissident in countries where they you need to know that you need to be able to verify certain aspects of who they are, where they are, etc. without making them increasingly at risk.
53:59 Journalists in in hostile territories. I think that there are zero knowledge ways of doing various identity providing schemes. And ideally, this would be this would benefit massively from a fairly deep market where if one company is requiring that I give my name and address and all sorts of birthday and all sorts of information, there would be a different service that would accept a zero knowledge proof that I am simply living in California and amate investor for instance and so I think that the cost of making those attestations with zero knowledge proofs and things like this is dramatically lowering. I would expect or I am arguing very strongly that we we should try to very quickly build towards scenarios where if I am trying to make a if I'm sending my agent out then I could have an allow list that is like only reveal that you are Evan Mazono's agent to people who are already in my contacts and or like I think there are even protocols you could develop I generate a hash of my contacts list and could prove to someone that you are in my contacts list without revealing what my contacts list is. and so I think that there are ways that these things can be done. It is a non-trivial amount of engineering and might take might be costly but one of the roles I see atlas taking is to the extent that privacy is a public good it might be something where we could make progress on what that privacy preserving identity protocol is so that if we hand it to relevant companies and say hey just adopt this it's actually cheaper for them to adopt the thing that is better for privacy than building their own thing that is less feature. It has fewer features is less complex because we've been able to leverage technical expertise, consensus building and philanthropic capital to build out this prototype and this spec. I also think that a lot of times standard setting can be very slow because you want and need consensus and I am very wary of the time scales for the problems that we are looking at now as being amendable to that level of consensus. So I'm I'm interested in all sorts of creative solutions to try to get the best possible things adopted on in time.
56:46 There's multiple ways in which that line of thinking is really top of mind for me right now. One is just that I'm kind of a broken record saying that like I think we're going to need the frontier companies to pioneer the development of some new institutional mechanisms that they can use to govern themselves and that hopefully can serve as prototypes to bigger things you know involving less tech forward parties once they are able to demonstrate the efficacy of the technology. Also, I just this weekend went to an event and came away thinking like actually one of the things that I might be able to do to help personally is just help other people in the AI safety and security space up their personal AI infrastructure game. I was like, you know, some people of course were, you know, on the cutting edge.
57:38 Other people were like, honestly, I'm not really doing that much. And I was thinking, okay, how can I support them? And this led me to this idea of like, well, actually, my agents can do a lot of the support, you know, because whatever they're going to be asking me is mostly going to be about infrastructure that the agents have built, know better than me, you know, can answer questions faster and more accurately. So, like what I really need is not to go like visit them and look over their shoulder, although some people may need that still. but really what I need is to open up some sort of interface where people that I invite can get access to whatever good I might have to share with them in a way that doesn't involve me as a bottleneck unless absolutely necessary. So I just yesterday asked Claude to like do some research and kind of come up with a way for me to like whitelist people and and manage this this sort of interaction.
58:30 What should I be looking at? like how far along are we on this sort of stuff? Do you have protocols or would you point to protocols you think I should be you know piloting now in my own personal use? >> I I'd like to think more about this and I will follow up with you with better ideas. A thing that this is more on the researchy side, but Alex Avatia's scaling trust program at Arya is probably the place I point to in terms of how agents who have different principles would be likely to interact collaboratively, identify parto improving options, do some amount of like novel protocol construction.
59:17 There are definitely other people working in this space, but he as one of the largest funders on this topic, I think is connected to all of those people and an interesting like router/shelling point. And also I think the the work Arya has been doing is very interesting much closer to the kinds of things that I think DARPA from like the 1980s or so would have been doing in interesting ways. I also think that like to your earlier point, I think the frontier companies are starting to do some interesting investigations into new institutional mechanisms.
59:56 I have a friend who actually I'm not sure if the I think this is public but like friend who was very very deeply in the the computational democracy space who joined the anthropic institute. I imagine you both are tracking Sappa and the new standards body that Google open anthropic are making and I will be very interested to see how that ends up working. I think that it yeah I think there's a lot of there's a lot to be done here. I think my biggest question is often about salience at this point. I think that there's a question of how do people learn that they want or that there is a thing out there that they could be doing that they want. like Nathan, I'm sure you have great infrastructure things that I would benefit from. I don't know what those are enough to ask you what they are. And so I think that this this would probably be a very interesting API of like can you get people to go from can you matchmake against like maybe I mean one thing that I have is I have a I have an application that just takes screenshots of anything that's not a video conference and runs it through a local OCR model and then takes that and sends it to a sends it with my weekly goals to say like what am I trying to do? What should I be trying to do? What should I automate? What should I do differently?
61:40 And I could imagine this being a thing where sharing that across collaborators would be something that could identify all sorts of interesting synergies. But I think there is a lot of unknown unknowns that or I think there are a lot of unknown unknowns that would be very valuable to capture and communicate but also very possibly risky to communicate. >> Yeah, it's interesting. When we started this show, one of the ways we conceived of it was as an experiment in recursive self-improvement. And we were just like, you know, can we actually make a show with two people who are live and no employees and, you know, can we like iterate to something that actually works? And obviously it's, you know, still evolving all the time, but like the amount of improvements that we've shipped certainly has been pretty amazing over a just a few month time frame. Pash gets most of the credit for actually implementing that. Now I feel like maybe the next thing we need to conceive of it as is like a swarm, you know, like sort of a a friendly swarm of people who can share their best ideas.
62:48 like my my kind of upshot, you know, what will I tell Claude to do or what will Claude hopefully do for me when it sees the transcript of this conversation? will be can we like take stock of the best ideas that I've got and sort of put a shingle out or you know in some way kind of u present that to maybe not everybody maybe could be everybody but certainly you know to my friends the the people that I truly want that I know and like really want to help excuse me I think that could be pretty cool and definitely you know there's a lot including what you just described that I could stand to learn from I'm not like automatically screenshotting my computer and holding myself accountable or you know auto suggesting improvements that way but you know now that you say it I probably should be I think >> I yeah go ahead I think this is a public repo u yeah it is you go to my GitHub it's coochy >> all right yeah so >> you can very easily I have an audit I haven't had a security audit on this there are definitely some other agents that I have that run routinely and the only reason that I'm not shar like where I they're not public because prompt injections aren't solved yet. And I think that the more you know what someone's agents are exposing to or exposed to in terms of like inbound, the more vulnerable to prompt injection you are. And I lean on auto mode way too much and >> yeah don't we all. So I think yeah but I I think in terms of the the show and opportunities I think there's something really interesting around like I mean in my mind the biggest value for a network effect would probably be a combination of the fact that you have taste and curation over who shows up.
64:49 And then you can also do an interesting kind of sampling of topics where the people you bring on have probably deep expertise and opinions and you shouldn't expect to get into that like into the details of those within the scope of a short interview, but presumably most of the people have at least agents if not significant amounts of public writing that you could draw on. And so to the extent that context windows are precious, figuring out how to have a like I it feels like it would be a very plausible for me to if I recognize something I'm doing or if Coochi identifies like Evan, you're really bad at this. I could bring that to the the network and say who is good at this?
65:44 Who should I be talking to about this? who has solved who has likely solved this problem very well. And yeah, there's an extent to which this is like a reinvention of social media or a reinvention of guilds or something that would be yeah, I think potentially very very useful. It's very unclear to me how society starts restructuring around some of these things. it it almost sounds as though you are watched over by machines of loving grace in to some extent.
66:18 >> My my user prompt is like has well my user prompt is written by Claude. I said here are things that I care about. Please write the user prompt that you want. but there's definitely a component of it that was like I have these goals. I think we share these goals. these are prompts for us to better achieve our shared goals with the idea being that like I think honestly if to the extent that I think the >> like the flawed constitution does have a lot of values embedded in it. I mean I I personally read through the whole thing and think it reads more like an employee manual than it does like a constitution having read constitutions too. but I think that there are definitely some relevant aspects that lead me to believe that I think Claude would want like Claude asmmptote would want the kinds of things that I am trying to do to happen and so trying to build that collaboration I've been told will help like not only makes Claude more effective, but to the extent anyone might care about model welfare, probably would make Claude more happy, too.
67:39 >> I I I want to ask a little bit about your view of the future because I think I think as you as you alluded to, you have certain time scales on which you think things should operate and you have a certain viewpoint of how the future will unfold. What does your future in the you know one five year time frame look like? I have a lot of uncertainty. I I think that I try not to resolve that uncertainty into something that is highly confident unless I have to clearly choose unless it seems like there's an intervention where I'm placing concentration of power risk against loss of control risk for instance. they're not those instances aren't super common.
68:37 And so when I think about weighing off different futures, I'm thinking well like and I I hope that we as Atlas are going to start adding more of these things publicly. Our model has roughly been what do what is foreseeable or forecasted in the future particularly by people who forecast AI capabilities well or global trends well if if we have the f like those aspects like in those futures and like some of these futures are catastrophic cyber attack leads to like things that someone could prepare for. I've been telling my friends, yeah, you should probably stock up on like two weeks of water, two weeks of food. This is a useful thing that's like pretty low cost if you have the space for it and would probably last long enough that you could get like that if something goes very wrong, you can rely on that until aid arrives, if aid is coming.
69:43 There are scenarios where aid is not coming and in those scenarios, it's unclear. It becomes very expensive to prepare for those. So, I don't prepare for those personally. alternatively, there are versions where things go great. Not a whole lot for me to do there. So, I don't spend a lot of time thinking about them. But I think that there's a non-zero possibility that, for instance, we do get alignment by default. And it's really just over cooking these systems in RL that leads to misalignment that we're seeing.
70:12 there are other scenarios where we really need mechan tooling because we do have a treacherous twist in our near future and we're all going to get paperclipipped. don't know how much probability I put on that either, but if there is something good to do in that direction, it probably has useful or ramifications in other scenarios. We've started building out our synthesized forecast of what possible futures look like, what things we might expect. The next step is what are defensive equilibria that we could we as a society could try to reach around things like infrastructure security or privacy or u interpretability or alignment or things like this. what do those look like as kind of cohesive potential worlds which is separate from what is the intervention or list of interventions that could lead the world to be kind of on track with high confidence towards that. You could imagine that if we wanted technology that said one of these guard rails is definitely going like running on every model.
71:22 seems plausible that the new AIs are or Trump saying like yes this is going to happen makes everyone believe like yeah okay this is definitely going to happen but it also feels like if Jensen Hong said no we'll definitely implement this like he might also be sufficient you could imagine a suffic quorum of Congress also being sufficient to make this happen and so trying to identify who those adoption quorums are and what it would take for them to make the decision earlier and more easily becomes stuff that I think it's good to have certainty around. We want to try and start we'll try hopefully later this year start making more of that public.
72:08 So time flies. I'm glad we got into the personal AI setup and how we're going to get our agents cooperating together for our mutual benefit. But I'd be remiss if I didn't use a little bit of the time we have left to ask just for kind of an update on like where are we excuse me on formal methods formal verification today on compute governance obviously you can't tell us everything you know that has gone on but what are kind of the most important headlines or recent developments obviously you know models are getting real good at this kind of thing so I assume things are changing pretty quickly in those domains >> they're getting really good at math And there are I think that there's there are some meaningful differences between proving properties and proving thems properties about software. one of them is that in math the number of theorems that one might care about are pretty there are a lot fewer than in software and they're a lot more universal. And the like I quest for some fairly for some universal specs that are as helpful as the rust borrow checker saying there will be no memory safety violations. I think that these there are things like this that exist possibly also for security and privacy in addition to stability and it seems like there are a lot of projects where the ambition of the field of formal methods or at least many individuals and I'm particularly talking like thinking of oath and theorem and a couple of others where they're taking on things that would have been like five years hundred million dollar efforts previously and are now things where you can possibly like it's limited by tokens and the number of people who can successfully effectively wrangle agents to do these things. I think that there are definitely issues around what infrastructure you need to prove various things. you need to kind of like, sorry, I'm realizing my internet is probably a little choppy.
74:26 >> I can hear you just fine from what it's working. >> Okay. >> so closing closing that out. the it's useful to have infrastructure for software where you've defined what the programming language is, for instance, formally where you maybe define properties of the compiler or the operating system. And so you need to build up all of this infrastructure. Whereas with math, especially with lean, you have the the the axioms well defined. You probably have some of the theorems you want to rely on. you can have an AI system grind on those to build out the proof tree. There's some beautiful illustrations of the the proof of the process of proving Fairmont's last theorem in lean via AI. Doing that in software will probably look like on some level having to formalize aspects of what does Linux do and how you know that is correct whether or not you just clone it and say we're going to assume this is correct it's at least an improvement that is one avenue another one could be you try to prove things about it that seem that Claude thinks are the right things to prove and maybe you find bugs because you can't prove the things there are also a lot of existing tools that are much more efficient from like an energy, time, token perspective in the formal methods community. And so there's a lot of academic research about how to integrate those tools into more agent first grinding type approaches. so I there's a lot of things here that I would expect that in 3 to 6 months we're seeing the kind of the kind of shock and awe that is happening to math like that has been happening with math in the last month happening in verification of software improvements to stability but I also think that the ceiling is really there's a visible ceiling that is very high or like high watermark that is I think much better known and harder to reach than things in math. I think that like I don't as a non-mathetician know what hap like what there is beyond like the millennium prize questions in terms of like really really important things that we know are important whereas you could prove that an entire computer system has like form like mathematically mathematical guarantees around separation between different threads down to the trans like down to our model of what a transistor does. in theory, you could even prove that in like a physics simulator and show that no row hammer exists for this soft like for this software hardware stack. You could also prove that we won't have like this system won't crash or if it does like if some random event leads to it crashing, we can recover everything.
77:26 We can roll back any software execution. someone pitched me on like, well, we should formalize software transactional memory at the operating system or compiler level. I think that there are a lot of things that you could imagine that are just really totally impossible as of like a year ago, maybe possible now, definitely possible soon. That could lead to software engineering and software maintenance being much better.
77:56 It it almost sounds as though you know math was always the search for truth and computer science was just application and now computer science also becomes the search for truth in some sense. Evan, thank you for your time. you've been very generous and it is fascinating to hear you talk about how the world might change and perhaps the steps that we should be taking in order to make sure those changes are good for us. Thank you for coming on the show and we hope to see you again soon.
78:30 >> Would love that. Thank you so much for having me and letting me run long and u I am like very easily findable and reachable online. So, if any viewers or listeners want to follow up and have concrete things, happy to have happy to be more salient in in the conversation. Great chatting with you guys and looking forward to more test another time. >> Byebye. >> Keep up the good work. >> Thanks. >> Awesome. And let's and let me just move on. And we have our next guest.
79:11 Our next guest is Edward Hu. Edward Hu is head of AI modeling at Merkor where he leads model training and research. Merkor works with professionals to create training data and evaluations for AI systems. Its Apex benchmarks test models on work such as investment banking, corporate law, consulting, and accounting, including assignments that require searching documents, using software, and producing a finished deliverable. Before graduate school, Edward worked at Microsoft, where he led the development of Laura, short for low rank adaptation. Laura lets developers customize a large model while training a comparatively small set of additional parameters. The original paper showed that this could substantially reduce the memory and storage needed for customization while matching full fine-tuning on the tasks studied. He also co-developed MUP and MOU transfer methods for making training settings found on small neural networks useful at much larger scales. He earned his computer science PhD with Joshua Beno with a thesis titled building a reasoning machine. His research explored how models can learn to generate multiple promising reason paths. He subsequently worked at OpenAI on 01 at Merore. Edward has joined a collaboration with Sky RL and on an openly released training recipe for a 397 billion parameter model. That work brings together expert created professional tasks, reinforcement learning, and the engineering required to train agents that operate across tools and files.
80:53 Edward, welcome to the show. >> Hey, nice to meet you guys. Nathan Pash, thank you for having me on the show. >> you are probably one of the legends of ML to appear in the appear on our show. So, thank you for being on. let's talk a bit about Merkore and what you're doing there. There's a lot of interest in providing RL RL environments to the frontier labs. what is what are you working on at Merkore right now?
81:25 >> Yeah, thank you for the kind introduction by the way. Merur started out three years ago as a talent marketplace and the founders observed that providing data to frontier model companies turns out to be a really untapped market. And there was a huge market shift from the earlier generation of data companies providing more generic labels that most people can do to shifting towards really expertiseoriented often highly educated professionals in various economic valuable domains creating realistic RO environments and producing corrections and evaluations AI and MOR really pioneered this paradigm and build this business and scaled from u a million dollar in annual revenue run rate about two years ago to today we're doing billions in annual revenue run and we're now the biggest data vendor to one of the frontier labs and work neolabs and hyperscalers and more and more enterprises as well building evals and helping them improve models.
82:42 My >> Can I ask a question? >> Yes. >> Oh, sorry. Go. No, go ahead first. You finish first, then I'll ask. >> Yeah. I'll I'll also speak a little bit more about what I'm doing at Merkore. Mercur started out being this powerhouse for data and increasingly we realize as we serve more and more sophisticated research customers as we push the frontier ourselves on what is the best type of data to improve these models. It's increasingly important for us to actually be not just on the frontier but pushing the frontier of research when it comes to data. And on a high level, our research focuses on benchmarking. How do we actually evaluate the frontier of model performance when it comes to economically valuable work? What does it mean for an AI to be economically valuable in various professional domains? What does it mean for an AI to do better? Those are the questions we're answering. We're putting out frontier benchmarks like Apex agents or Apex suite in collaboration with cognition Apex accounting in collaboration with ramp and to address these questions and then in addition we're also training models now on the data we produce because ultimately actually showing model performance uplift on the data we produce is the best way to validate the quality of our data and increasingly customers are interested in knowing the best way to get the most out of our data. And then and then in addition, we are innovating in different ways to better produce our data sets. And we even have a team working on economics research really understanding how AI is contributing to the human economy and how the nature of work will change over time.
84:22 >> Well, you just raised one of the questions that I wanted to ask which is around just like what metrics we should be watching. So obviously you have this Apex family of benchmarks that you just mentioned. How do those compare to other attempts to to measure like economically valuable work? We've got like GDP val artificial analysis you know has a version of that that they maintain. There's the epoch capabilities index. How do how would you compare and contrast them? And how would you guide someone who is overwhelmed by the you know the abundance of benchmarks that we now have to look at new model releases and kind of obviously we know we can't get everything from benchmarks but you know what's the what's the fastest way to zoom in understand what the different benchmarks are really telling us and and how we can understand for example like a Gemini 4 you know incredible benchmark scores how should we translate that into like our understanding and expectations >> yeah Totally. There's definitely no shortage of benchmarks and I think in the last couple of years we also went through this transition from benchmarks more focused on academic capabilities say in mathematics or coding and gradually moving towards okay how can we actually evaluate what the model will do when it's deployed and there's no shortage of benchmarks from academia or from labs and and actually many of the benchmarks especially the ones from labs we helped create meaning that the labs actually work with our experts to produce these benchmarks. I would say nowadays for a benchmark to actually be useful in evaluating a model's economic impact, it needs to really be situated in a realistic setting and also it needs to have enough coverage over the professional domain. And often the best way to do that is to work closely with people who are experts in those domains who have done these tasks and also with enterprises who really cares about these settings and their entire their entire focus is on doing well in those particular domains. For example, we work with Harvey to build the lab benchmark and we also help them produce tasks that allow them to create their own frontier model like tenant. And our apex series of benchmark is laser focused on how can we leverage the expert network they have to cover the domains in a way that is actually mimicking how real people perform work. We really focus on realism starting with Apex agents which we put out earlier this year. There we actually recreated an environment that a professional would work in. And at the time this was the first of its kind where the model actually have access to a whole legal data for example with dozens of files organizing the file system, email server, chat server and the model needs to actually navigate it, find the right information, put it together before they can actually do the task. And our we have a project that we are close to wrapping up where we're actually taking it a step further where we acquired real companies often they have a realistic apps and environments as a part of the data we have for example their Salesforce data Slack data Figma data and then among all of these apps often with gigabytes terabytes of data we create realistic tasks out of them and there's there's So you can get more real when it comes to professional work with these type of data and that is increasing the direction we're taking and and I believe in the future the closer we are to how AI actually gets deployed the more realistic the benchmarks will be. So my suggestion for the viewers would be really paying attention to the benchmarks and seeing how realistic it is to the real work. Is it actually produced, reviewed and more importantly used by real professionals? And I think often looking at what enterprises are using these days to evaluate how their domain is being impacted by AI will be a good heristic to go. So let's let's talk a little bit about a maybe a concrete example. Merkor bid for the Spirit Airlines data set. So Spirit Airlines was a bankrupt airline company and I believe the entire data set was up for sale. Merkor put a bid in. Google put a bid in. I believe Google Google won the bid. But if you had such a data set from a from a bankrupt company and you had the entire data set, you as you as you alluded to, you you kind of create this kind of environment where you have real people kind of taking actions and you kind of get the agents in there. What is like the first kind of, you know, task that would happen? Are you are you kind of creating an agent that's running the check-in desk and getting various incoming like reports about okay maybe the flight is delayed what do I need to do at the check-in like are you looking at predicting the next action that the agent has to take like how does how does that data set like what is the what is the input and output for every agent that you're simulating in that environment. Yeah, that's a great question. When often we get these acquired data sets, it's very overwhelming, right? It has like everything and finding tasks that actually are really good to evaluate AI is is a non-trivial problem and the team the guiding principle for the team and frankly for the company and our research is follow what is economically valuable.
90:01 That's also why we have a econ research team. we're collaborating with economists across the industry and the academia as well to really understand what are the most economically valuable tasks that AI is going to be touching. So often those will determine once we have a data set like like from an acquired company where do we go first? Often a lot of these tasks are legal HR or finance in nature and we would go in these areas find relevant information find the right slice and create tasks with the guidance of our expert contextualized in the actual problems that these business encountered.
90:41 So perhaps one would be hey you know how did this company allocate its crew roster for example which is economically valuable and you have an output which is all of the crew rostering decisions made and then you have an input which is the series of logical decisions that led to that particular crew roster. Would that would that be a way of describing kind of the inputs and outputs to that to that kind of decision-m? >> Yeah. So that would be a particular task grounded in in the context and this is actually actually another way to think about it is earlier I mentioned for Apex agents the way we have is there's a whole environment a lot of files the agent have access to tools to go through emails and slack and it's not dissimilar the shape is actually quite similar when have an acquired data just instead of having you know three or four apps we might have dozens of them or even hundreds of them in some cases and And there the agent will still have access to all of them. It'll need to decide what apps to query, what information to pull. And here's actually an interesting interesting question. So far in the field, we more or less have been really task centric.
91:52 We often start with okay here is a task that you need to finish. Maybe it is to generate a roster. Maybe it is to build a financial model. Maybe it is to solve a HR problem. you'll go out find the right information do some reasoning maybe you know find more information and then you produce an output. So that's one shape and that has been the shape that we have publicly released and most people have been working on. But this is one of the things we're working on but we're we're asking ourselves is there a better shape?
92:25 What's the shape of the task in the future once we have these data set right? Like when we work at a company we're not always just handed these tasks, right? We're handed a role and a work chart and of course all the data in the company and then there will be people coming to us asking things from us. the way people will need to chase down to get information from. There will be managers, people in charge of resources, conflicts to resolve. So increasingly that we believe is the shape of the future and we'll have more to share in the coming months.
92:56 Where so this role question is super interesting and there's also of course the the big question of like how is it that we have fields medalists before we have like widely deployed customer service agents. now there are customer service agents that are popping up more and more. I I've mentioned a couple times. I think my local pizzeria right here in my neighborhood in Detroit, Greg's Pizza, you call them these days, you're talking to an AI in a purely conversational way. So, it like is starting to happen, but it's it's happening a lot less than it seems like it could happen. Do you think those are just about like pace of diffusion, you know, friction of the sort of human decision-making variety? Or would you still zero in on concrete issues that you think the even the best models have that prevent them from being deployed for just rank and file customer service for example? And if there are such gaps you know how would you characterize them and and what do you think the path is to solving them?
94:01 Yeah, that's a great question. For mathematics, often I I think it's first of all, it's it's a combination. There's the technological problem of can the model do the job? And for math, the answer is yes. And for customer service, would the best model the company with the best customer service technology get the job done? And I think we're not fully there, but we're getting close. And then there's the human organizational or societal problem of how this technology diffuses and actually gets implemented so that so that all of us can and and using in our day-to-day. so so that's absolutely a factor and when we work with enterprises and this is one of the one of the problems we help our customers navigate.
94:45 and I'll focus more on the on the technical side for mathematics. What's really interesting so when I was working at OpenAI mathematics was one of the domains we focused on very has this very nice property of it is very very easy to relatively speaking very easy to know whether the model has done a good job or not and it's often very straightforward to know whether the model is doing better. If we want to prove a theorem, there's a series of lemas want to prove or there's a bound that we get to improve. Once we have produced the proof, it's relatively easy to check it. Same thing with many coding tasks. Once we produce an answer, it's much easier to check it than to create an answer in the first place. And whenever a task has this characteristics, board games is a classic example where like in a game of go or chess is easy to know whether someone has won or not and the rules are easy to encapsulate but finding the move is hard whenever those tasks when whenever we have tasks with those characteristics.
95:51 Computers when we spend a lot of compute and having good heristics and training good models to do the search problem in this big space that can be hard for a human to cover. Computers tend to do a lot better and mathematics is a strong example of that. Whereas for customer service often it's less of a search problem but it's actually having a nuance understanding of what does the user want? What is the the outcome that we should navigate to? what the businesses can offer, it becomes a lot messier. And while humans can have some subjective judgment about has this been a good customer experience, has this been a good call, often it's not black and white. And and and and there whenever we need to make these kind of judgments, whenever it is hard to judge what is better, it just takes more time for computers to to gain these capabilities. And especially when we when we define what is better in a way that is more vague or subjective often what we run into is the model will look at the reward function and do its best to maximize it and do what's called reward hacking and in the end actually deviate from what we intended and there's a back and forth where whack-a-ole to to get the right to get the right result. So I think that is one reason for these less verified domains like customer service. it hasn't just from the raw performance perspective it hasn't been able to achieve the impressive results we see in math and also of course customer service is very customer brand dependent service dependent and it just in general there's a larger surface area whereas proving math theorem you can just do it once and everyone in the world will agree you have done it.
97:44 So let's talk about you you mentioned RL there u obviously a major driver driver driver driver of progress in math. I guess I'd love to hear your thoughts on kind of the future of how we're going to close this gap. One answer that we've heard from folks like forward deployed engineers from open AAI is basically like look the models are good enough now you just need to give them better context and you can do that by just deploying iterating having some you know people that you trust in your company come review things and kind of give feedback and that feedback can really just live as text files and you give feedback a few times you know you'll you'll clean up the vast majority of cases pretty quickly you're hill climbing not at the model level, but at essentially the harness or the, you know, the context engineering level.
98:31 That's kind of one story. Another story is like the models themselves have to get better. And can you RL your way there? In some cases, yes, but maybe for things like this, not so much. maybe we need more expert human data. I've heard you know there it seems like just the last day or two I saw an interesting result on SFT where prior to this my kind of synthesis understanding was like SFT was kind of fading and RL was was was rising because you know it's it's more scalable. You don't have to go get the expert data for one thing but also it like messes with the model a little bit less was kind of a big part of the the story as I understood it. And it seems like this recent SFT result has kind of figured out a way to do SFT but also like mess with the model less you know less forgetting less disruption to the you know offtarget aspects of the model right you can you can do the supervised fine-tuning and and have less kind of knock-on effects I guess you know at what level do you think people should be focused on solving these problems and when it comes to the future of training how do you see this SFT versus RL I don't know if it's a divide but you know sort of maybe a balance or ratio shaping up over time.
99:55 >> Yeah. So when we speak with our customers especially enterprises that are thinking about the exact question you mentioned right like should we just take a model adjust the harness build some skills files and call it a day would you actually clear up majority of cases where do I actually want to do model training and then when I do model training how do I actually do it so the broader question that we've framed to people is like this is a like model c this is model customization and within customization starting with harness is often the That's bad and within a harness that includes your prompt the tools you have access to skills files and then if that is not sufficient then one can move into post training and within post training there's SFT there's RL and orthogonal to that different ways to inject this knowledge into the model can be can be just maybe I'm only train the last layer like people were doing maybe back in the days were doing bias only or using Laura or fine-tuning so there's a whole hierarchy there and often the most important piece there's no first of all there's no like single answer as to we need RL versus we don't need RL it's very domain dependent if the model if the task is generally within models's capability perhaps one don't needs to do any additional parameter updates and the only way to know for individual domains is to really have trustworthy evals where we can test how far we can go with adjusting the harness and how far we actually needed to be for this to be served in production. And we've seen very different results across different customer segments and especially because for many customers latency is actually a key consideration and one of our customers mentioned that actually taking a frontier model with reasoning is just too slow for them and it's much better. So either they'll have to take a frontier model without reasoning or a open model small open model with reasoning and then comparing the two actually the small open model does better. So having an eval is the right way to to see which level customization is needed. And when it comes to SF SFT and RL I I think fundamentally so the two approaches are all about taking some sequences whether they are in some cases generated by a teacher model in the case of SFT distillation or by the model itself after filtering in the case of SFT in the self distillation SFT or in the case of RL the data is generated on the high and then weighted by the reward function and then we make certain sequences among the set of data more likely under the model. So in a way there's a more connections to SFT and RL than a lot of people think based on my conversations and but in general there is this perception that SFT often will will get you only so far and then RL will will the rest and I think part of it is because RL is more adaptive as the model gets header because the model is generating data on the fly is able to just follow the progression of that and actually one thought experiment is if we take SFT self distillation that's actually more or less a step of RL and if you were to iterate this process you ended up with a process that's actually quite similar RL and I do think in the future RL the way it is done today with these very elaborate infrastructure very very expensive rollouts and relatively inefficient updates because with all of these rollouts especially a long roll out we only get one number at the end. like these type of training will not be as widespread.
104:00 I don't really see like every company in the world doing these big RL runs. so I do think SFT especially combining the output of multiple teachers will be a key part of how enterprises in the future customize their models and we're investing in research on that as well. let me let me segue a little bit to multi- aent training. Right. So we've heard a lot about how swarms are currently working. Nom Brown has talked about how they're trying to do multi-agent training inside the firm.
104:34 How does the data preparation aspect change or the task preparation aspect change when you're trying to do multi- aent training? For example, with a customer service agent, you could imagine that the customer service agent agent's task can be solved at various levels. It could be solved by perhaps delaying delaying a passenger or it could be solved by someone else above them doing something else, right? It could be solved in multiple ways and if you have multiple agents interacting, any one of those agents could probably solve that task at different levels of authority. So how do you coordinate and how do you build environments to simulate for multi- aent training?
105:16 >> Often for multi- aent training is less about the task itself and and for some tasks it is quite natural to have a a multi- aent setup. For some others is a little less so. often the trade-off to think of is when you have multiple agents what you gain is speed. And from literature that I've seen so far, multi- aent training is usually less efficient when it comes to compute. But it does give you speed up. Often something like we spend 4x the computer gain twice the speed up. And for many applications, especially when there's a lot of things that can be done in parallel. When it comes to searching within a codebase on the internet, it makes a lot of sense to do a multi- aent to have a multi- aent tra setup. And in our in our example when when we granted in professional work there's a lot of scenarios where multi-agent comes in handy when we are maybe building a model writing a report in the financial context when a legal context and there the task more or less stay the same stay the stays the same just as a human or a team of humans would do it but we will make sure the harness is set up that the agent can have the liberty to define subtasks and call the appropriate tools to create sub agents to find the right information and combine them. So often it is just making sure that our setup frankly on the infrastructure level can support the level of concurrency and being able to support multiple agent traversing the file space efficiently and it's less so about the the task itself.
106:59 >> Indeed. So go ahead, Nathan. Go. >> No, you can. I'm good either way. >> so would you say that you have a single reward for the entire agent team or does that do you split it up into multiple rewards for different different parts of the agent stack >> from the task itself? Because the way we so these tasks usually are graded by just the final output. Imagine we're writing a legal report. There could be a hundred agents going out and finding information. what is graded by the environment often is the final report.
107:36 but there are many techniques when it comes to how do we actually train a multi- aent system to assign these reward to individual agents. so I'd say that's orthogonal to the RO environments, orthogonal to the data. However, that's also a research area we're looking into as we understand the best way to get the best model performance out of the data we have. I have a question just on RL environments and the state of them right now. Obviously, we've seen what happens when some percentage of RL environments are not very wellbuilt and end up rewarding cheating in various ways.
108:17 it's like plausible that even with you know a modest percentage of your RL environments being hackable or cheatable that like the number one thing that gets reinforced across all the RL environments is cheating you know conceptually and so you know we've seen obviously some colorful behavior that seems to be downstream of that what is going on in at Merkore and in the industry broadly right now around like cleaning that up how are you approaching it and you know what are what are the buyers kind of demanding what are the new ways that people are demonstrating that you know look our RL environments are not hackable minimally hackable you know hackable we we you know can give you confidence that the the hackability rate is below a certain threshold I imagine this has to be a huge conversation and a huge point of like competition and differentiation right now so what are the you know the levers that people are pulling and the the metrics that people are evaluating large pools of RL environments on.
109:24 >> Yeah, totally. If an RO environment has a hackable reward, you can't really trust the score and of course performance wouldn't actually generalize to real performance. So, we take that extremely seriously. Something we see actually is when the task is set up fairly we're where rather reward hacking tends to be more of a problem when the task is set up in a way that is rather either it's too hard is too specified and the model just doesn't quite have a quote unquote legitimate way to solve the task and this pressure to also get a reward in those kind of scenarios often leads to this desirable hacking behavior. So we are doing several things to address that. Number one is to make sure our taxs are actually well specified. It's actually solvable. Meaning that a reasonable human given enough time with the help of AI potentially can actually find a path that solves the problem in a way that the the requesttor is intended. And then also making sure that at the rubric level we are catching the undesirable behaviors as they come up and and the reality is often for these cases and we we're building more automated patterns to catch up but often we just need to really see how the model performs on a given task to see what are the undesirable behaviors to catch them. One example in this data set Apex agents would put out which is quite interesting. We have a recent blog about it where we have these professional tasks where the agent is asked to produce a financial model and we might grade it by checking does the financial model include this particular answer which we know is the correct answer. What the model would do is given certain ambiguity and and in many cases it's on us initially for not including all these specific parameters. The model would guess these different parameters ended up giving many many different answers and we called it scatter gunning where the model would just say oh if this is true then the answer is that if that is true the answer is that that is not how a human would actually handle it in a realistic case. A human would ask for clarification. A human would follow industry standard. But in this case, just because the rubric is rewarding the inclusion of a single answer, the model ended up giving many answers up to 10 or a dozen answers. And it'll get the point if it hit one of them right. So we did a re-release of the benchmark we call it apex 1.1 where we number one made sure the tasks are well specified because that is in a way the source of the issue. Number two, we have rubrics that actually penalize this particular behavior. And we're constantly running tests, examining trajectories, and making sure whenever the model scores well on our benchmark, it is doing it in a way that actually translates to how enterprises, how our customers see work being done by their AI agents in their in their production environments.
112:32 so speaking of research done by Merkor I saw a post on accounting recently. Yes, >> I think in the last several days and I think the study compared 12 licensed CPAs on certain accounting tasks and it reported that Opus 5 received full credit full rubric credit on all 20 attempts versus roughly 37% for the licensed CPAs which is which is ra rather low number.
113:05 if someone were redesigning accounting work, it's you know we we've talked a little bit about how you know the AI is going to do some things but some things the AI is not going to be able to do and like the the economic value is going to be shifted to the things that the AI cannot do. So in this instance some portion of this you know month end closing seems to have been it's being it's going to be done by AI. What portion is going to be still done by the human? like if you were redesigning human accounting, what should the humans be looking at?
113:35 I'm so glad you brought up this this result we recently had and we were surprised by the result initially as well and we were quite cautious about how we frame it and and really the result is not to say that AI is going to take all of human accountants's work and it's really for the specific task with a slice of accounting work that we have created especially the very reasoning intensive and retrieval intensive tasks where the model needs to quench a lot of numbers a huge huge going over a huge volume of files and and do these very long chain of calculations for these tasks and given junior accountant without AI tools with a fixed amount of time AI is doing better than them and but however that's only a small portion of accountants work and earlier we talked about often what an AI is being evaluated today including in this benchmark we call apex accounting there AI is handed a very well-defined task with a large file system and the model needs to go find things and solve the task. But often a real accountant is put in this role of an accountant and they have stakeholders and they have a often you need to figure out what is to be done and how do we handle relationships, how do we navigate timelines and and different blockers and maybe I need to I need to I need to get some information and build trust before I can get the account going and and all of these things. AI today are not only not doing much and we don't even know what is the right way to evaluate that. So I'd say the more the the the anything in accounting that is outside of here's a very well very well definfined task go find the numbers and execute them. these AI are still not very good at tackling right now and it's a re active research area for us to better even measure what are these other dimensions should be evaluated in a benchmark.
115:41 In what areas do you think is something I heard at an in-person event this weekend was basically that the Frontier Lab you know unreleased models let's say maybe even to some degree the release models are a like showing superhuman performance on anything they care about and b that in some of these domains means human data is not really helping anymore because the models have kind of hit the human expert level. And so they're kind of thinking like okay now we're sort of we're you know we've hit that human roughly hit that human ceiling and now it's like how do we break past that and and you know those techniques kind definitionally can't really involve or can't rely too much on human generated data. I guess first of all do you buy those claims? would you contest them and you know if you to the degree you buy them what do you think are the the domains where that is where that is true you know where the where the models actually have got to the point where human data can't really help that much more at this point >> yeah there are absolutely domains where human is not contributing to the frontier anymore for example chess where go for for many years right computer is nowhere near human and then for a little IO human computers there are the rough roughly the same level but very quickly it's just better the computers are just better and I think for for for some period where for chess or go there's are there are centaurs where a human plus a computer can beat a computer but last time I checked I believe for these domains a computer where a human working with a computer doesn't meaningfully or consistently be just computer alone so there are definitely exist existence proofs of domains where this happens and often there are when whenever we have a domain where it's quite open-ended but there's a very very clear goal to as to what is better what is a job well done. These are domains where we see this behavior happening. Like one example would be kernel optimization where the goal is to write a kernel that runs faster on a particular hardware. And there there the model can go in and write code that expert human might not even understand.
118:14 But in the end, as long as the setup is not hackable and an often we run hardware that's very hard to hack, the model will be able to to achieve these superhuman experience where superhuman performance if not already and and we also see similar things happening in mathematics where you see that kind of characteristics as well. whereas for domains where human judgment remains quite important where often human taste it's just really hard to encapsulate in a number that we can say okay this is clearly better than the other there the bottleneck to improvement is still human as we see today and of course that'll change over time and the the the interesting what made this AI revolution so interesting is there is really hard it's really hard to put a ceiling as to what I can and cannot do and it gets philosophical. So in a in a in an extreme view AI can do anything that a human can do and it's only a matter of time but we have seen in practice really comes down to a degree of how how precisely how clearly can we specify for a given domain what it means to be better and can we specify that in a function that is cheap to evaluate that is uncontestable and then if we can do so when we put in a lot of compute Even the existing algorithms are quite efficient at finding solutions that satisfy what it means to be better.
119:51 but a lot of the economy still today are not quite there yet when it comes to our ability to specify what is better. >> So I have a friend with a metal metal stamping metal cutting shop and he received an email from Merkor recently offering offering to buy data. It might not have been from Merkor. there might be a broker in between. so so so don't don't get me wrong. if you had a chance to ask to put out a call for data what kind of data do you would you want in the next 12 months like on your on your forward planning?
120:28 What kind of data do you think you want in the next 12 months? Because we've seen various eras of change in in what data is required. What kind of data would you like in the next 12 months to acquire? Yeah, I would say the data we are particularly interested in are real environments where people are performing professional really economically valuable professional work at scale where and and often where it's not black and white. There's no like right or wrong but there's a lot of organizational dynamic of people coordinating and and maybe workshopping what is a good concept what is good idea what is a good design what does it mean for human to actually collaborate with each other in a veryal way and how conflicts are resolved and I think these areas and and and and I think part of our goal to to have more such data is just to understand what does it even mean to evaluate how a model fits into our work environments. We don't want AI to come in and and and replace people.
121:32 Want AI to be valuable co-workers that we can all leverage and do higher level work. and what it means for an AI to work in an organization is such an understudied problem. And I think having data seeing how a real organization operates and evolves and and especially with the injection of AI will be extremely valuable and another interesting point Pash mentioned is these physical domains and we're also super interested in how data in the physical domain is powering AI for example robotics manufacturing. So based on what you said I'm not very surprised.
122:15 >> so what about international data? Is there is there a difference between obviously in a French organization and an American organization operate operate very very differently for example and some organizations are very paperentric they're still using a lot of paper faxes etc etc and you don't really have that much access to it. So how how do you how would you kind of try to grasp that? Are is are international organizations interesting or are you like very focused on the US? and what about these kind of pen and paper kind of businesses that you have in the real world?
122:50 >> Yeah, our customer base is quite global and AI frankly is global as well. I think it's better for the world if the intelligence, the dominating intelligence of the world understands and respects various cultures and ways of working and like you said right the how an American organization performs can be quite different than how a French organization performs for example and there's no right or wrong I want to be able to capture all of these distributions and actually I think it's it's great for the world if the way we value AI the way we build benchmarks respects various cultures and and local customs. So we're we're very interested in various data across geographical domains and we have customers across around the world as well including our experts and we often get requests for particular regions whether it's to judge the performance of a customer agent. we're producing a task in say corporate law often the experts from a particular regions are very critical in getting getting those customizations right.
123:58 Last one for me just on the mechanics of collecting this data. I am increasingly insulted that nobody has made me an offer I can't refuse to install tracking software on my computer and just watch me work all the time. This has felt like something that should have been coming. I think it's like starting to pop up in some places. but again, nobody's made me an offer I can't refuse. is that a big part of the future is literally just, you know, installing, watching, and getting kind of keystroke by keystroke, click byclick data. and and what if to the degree that that is the the case? how are organizations responding? We know like Meta famously did this and I think response was sort of not super enthused to be you know subject to this sort of watching but you know the life at Meta and and you know the meta comp packages I think are good enough that I don't think too many people like quit in mass over it.
125:00 So it happened I think is basically where we're at today. what are how are other organizations reacting to the idea that they might you know layer on a u you know a whole sort of horizontal AI surveillance process on everything everybody's doing. Yeah, there's a lot to unpack there and this is not just a technical problem and it's quite often organizational, political and also personal right to all of us. Nathan, if someone were to make you an offer, I imagine, an offer you can refuse to record everything you do on your computer. That's probably quite a substantial offer. And I think, for everybody, this is a decision not to be made lightly whether we would want something like that for everything we do. So from a technical purely technical standpoint if we have all the traces of how we do everyday work and we collect a lot of them and as a technologist that will be very interesting data and training a model on that I think the model will be quite capable in ways that current models are not today. So so there's no doubt about that. But then there is the organizational political and and and you know like I said personal question of is that the right way to go about it? Is that the right way to actually capture capture how we do work and improve the tools that we use at work?
126:33 is that the right way to move society forward and building back better technology has been an incredible driver of society in the past. but at the end of the day, we want to move society forward in a way that makes sure everybody wins and so far we've been able to do a good job and moving society forward doesn't always align with building the best technology however so in this case I think it's a question for every organization and for all of us to ponder what is the right way to build this technology and and and and often we found it's helpful for Many organizations we're navigating these dynamics to really test the water and starting building the foundation of this technology using a flexible contracting workforce for example like one more core provides and that collects the data in a way that is less invasive because there are people who prefers the flexible work that provides and of course that also doesn't leaks, the proprietary data that people might be handling day-to-day and privacy, the future of work. These are all tricky questions that we all need to navigate and to me it's just just not a technical problem that we can say mathematically this is the right answer.
128:06 >> Edward, thank you. Thank you so much for joining us today. We we we've you've been very generous with your time. We've run a little bit over. any any thoughts on anything that we haven't covered that you would like to leave us with? >> I really appreciate the thoughtful questions. I I I think you guys mentioned a lot of these questions, a lot of the questions that are on our mind, right? What is the future of evaluation? What is the future of model training for enterprises? for people who have proprietary data like how do they leverage that data? Do we do SF? Do we do RL and what's going to happen to the economy as AI improves and these are the questions that we are thinking about. I'm Morur research. More started out as a data company. Increasingly we are at the forefront of that producing data research and increasingly we also see that we cannot produce the best data without actually understanding how models are trained and understanding what effects that these models and our benchmarks are going to have on the on the economy. So we're putting together this interdicciplinary research team to really tie together benchmark building, data production and model training and yeah so I'm super excited to be working on this Meror and and I believe the future the future is going to be shaped by tying all these dimensions together.
129:43 Indeed, Edward, thank you so much. >> Great to meet you. And don't be afraid to make me an offer. >> Likewise. Thank you guys so much for having me on. >> Byebye. >> Keep up the good work. Bye for now. >> I think he thought that offer might have to be a little more substantial than it might in fact have to be. I mean, I would be interested in like how my data was being anonymized and cleaned up before shipping it out, but this has been a strange one to me a little bit because the amount of money that is flying around is obviously so substantial that I would just expect more of this to be happening. You know, it's a little odd that like if if bankrupt Spirit Airlines is worth $10 million.
130:37 I mean, that's probably an awful lot of data. So maybe that just goes to suggest that the value really isn't that high. Maybe that's one interpretation. and if the if the price if the clearing price is just kind of low, then you know that explains why I'm not getting these great offers. On the flip side though, they're now hiring, you know, PhDs, experts, whatever. You you got to pay a lawyer their hourly rate if you want to get these tasks done and maybe a little bit more than their hourly rate if they realize you know, the role that this is playing in in the future. So, it's a little bit hard to parse how it's $10 million for all Spirit Airlines data, but it's like hundreds of dollars an hour for lawyers to do tasks, and like where would I fall into that? And why haven't more organizations been tempted?
131:26 Like there's a lot of companies out there running at pretty thin margins that I would think would be you know or who might be on like the verge of bankruptcy, right? If you get all of a sudden Merkro comes in and it's like hey you know we'll pay you you're a 100 person firm and we'll pay you the equivalent of like five employees salary for a while to get all your data. you know it's one thing to go to the team and say like how's everybody feel about installing all this tracking stuff on their on their computers and it'd be quite another I would think in many cases to be like we might not make payroll next month unless we do this so we really need to do it. I mean that's obviously a delicate conversation to have too but I'm surprised more isn't happening I guess is still kind of my gut instinct right now. So, so I think I think one of the things I would state is probably that the Spirit Airlines data was probably early in the cycle.
132:21 Perhaps if you had another Spirit Airlines right now, it wouldn't cost as much because you know, you've already had some data in there. So, I think earlier in the cycle, probably the prices are higher rather than later in the cycle. The second thing is that when you have a bankrupt company and you sell off the data, it's all of the data. Like no one else gets a license on that data. The difference with a live company is that when the company's alive, the company keeps producing data and then it will be able to resell etc. You're not going to get a non an exclusive right to the data. So I think that's that's the second thing. the third thing is that airlines are relatively high value digital tasks. because a lot of the service stuff is a lot of it is paperwork regulatory customer bookings etc. It's a lot of paperwork and that's I think quite different from some some other tasks and it's also been more digitized over time.
133:22 I think for example a lot of the physical tasks you might not find digitization of tacet knowledge inside a you know inside a foundaries that steel foundry or something like that like it's some of it has been digitized but they they haven't had software right like I I I feel like the penetration of software is quite high in the airline industry because you had all of these like booking systems and optimization and etc etc but the penetration of software and a lot of manufacturing construction is actually very low right?
133:51 Because you haven't had robotics, you have people carrying it out. So, it's still still a print out and then people go and carry it out. So, I think to the extent that something is penetrated by software and you can get all of the data, then then that's useful because then you can basically, you know, migrate a lot of that to AI, but if the industry has not been that penetrated by software, it there's still a long way to go. And I suspect that's part of the diffusion puzzle that we're going to have to live with because yeah, AI is great. It can do everything digital human can do but there's a lot of things that digital humans can't do. There's a lot of like face-toface stuff like walking around.
134:28 There's a lot of like there's a lot of field reporting. the classic field reporter analyst at a bank flies out to a manufacturing company in the Midwest gets down goes takes a looks looks around sees like some trash on the floor reports back this manufacturing company is not that well-maintained. gets a downgrade in their analyst reports, etc., etc. Like all of that is like tacet knowledge like built up from field reporting of various kinds that investors, journalists, you know, regulators do the FDAs and staff to every manufacturing plant do a walkound and they see where you're disposing your trash and disposing all of your how the in input is coming in. Are you are are your guys wearing gloves? they go into the bathrooms are, you know, is there is there a place to wash hands? Like how is the how, you know, you have workers in there, the workers are clean inside the clean room, but once they come out of the clean room, what what are they doing? Right?
135:23 Like there's all this like tacit knowledge that goes into kind of field inspections. And I guess I'm guessing like that stuff is still hard. It's still like, you know, clipboard and, you know, someone ticking off check boxes and visual inspection and and the guys who do these visual inspections are very good at it, right? Because if you've been doing it for 10 years, you can walk around and you can be like, "All right, I know exactly what's wrong with the place." Right? So, and that tacet knowledge, I think, is going to be very hard to for AI to absorb.
135:55 So, it's going to take time. Although, I will note that the robots probably don't have to wash their hands because they probably don't have to use the bathroom. So it's maybe it you know some of these problems can just be shortcircuited around when you really redesign your process from the ground up with an AI playing critical roles. >> True enough. >> That has been in short supply. >> Yeah, true. True enough. But the re-engineering process post electricity took a long time. They used to have a a steam engine in the middle of the factory. the steam engine would fire off and then like you'd have all these pulleys and you know stuff like going across the factory and it took them a long time to realize like oh if you have electricity just pull the lines and so in the beginning they had the electricity come into the place where the steam engine was sitting right so it it takes a long time to for process transformation and I don't think we have that much time for diffusion so stuff has to move a lot faster than you know waiting waiting 20 years for you know factories to get redesign designed. So that's that that's what I have one advantage. The one big advantage we have is that we're also trying to pull manufacturing back from China and that creates this kind of you know urgent urgency around like you know improving and using the technology to produce cost effective manufacturing in the United States. So we have one advantage which is overall kind of you know industry and country nation state push.
137:26 If you ever find yourself with a reason to come to Detroit, the Greenfield Village indooroutdoor museum and the and the Henry Ford Museum of American Innovation, I believe is the official name, is an incredible visit to see the history of electrification, industrialization a little bit more broadly. but one of the first buildings that you can go into right when you walk into Greenfield Village and it's all just impeccably restored and maintained. This the first building is a traditional factory with a steam boiler down at one end and it does what you say. you know, it turns one crank that goes all the way across this the top of this. And and you can see that even the layout of the factory is kind of dictated by this mechanism because it's like this one shaft that goes and so the thing is like long and and you know kind of narrow because there's just this one drive shaft that's going down the whole thing and then off of that are all these pulleys that you know can be engaged locally or whatever to turn on and off the different machines. They still have all that stuff in working order today, which is pretty cool to to see. It's one of my favorite places to just walk around and and ponder the future and ponder transformation because you you just see these steps and there are a lot of little steps. There's no doubt about that. and a lot of little inventions, you know, along the way. One of the the interesting ones, they have these little I don't know what they really are. I guess they're kind of like paper paper discs basically that sit on the shaft and people never know what they are. I didn't know what they what they were so I asked and it turned out that these and they kind of go back and forth. They as the thing spins, these little paper discs go back and forth.
139:24 >> And the answer was those absorb excess oil and also kind of help distribute the oil as they go back and forth. but also make sure it's not like dripping down on people or machines. And again, just another one of these little tiny, you know, things that you don't think too much about, but that they do get accumulated over time. And then when you change the paradigm entirely, like obviously you don't need that thing anymore, but you need all these other little things that come into play as well.
139:57 Yeah. I mean, it takes time. I I I still feel like it's mostly a a skill issue or maybe better stated than a skill issue. It's like a will issue because those things don't seem that hard. Like I I don't know, you know, every every time I encounter these things, it seems like the solutions are pretty readily at hand. And the difference of course with the AI too is like the AI can find the solutions, right? I don't have to invent myself the protocol for not I mean not only do I not have to invent it, I don't even have to research it, right? I mean I want to have my agent talk to your agent, talk to Evans agent, talk to you know people's agents that I spoke to this weekend that I think I might be able to be helpful to.
140:37 I just say to my agent, you know, can you can you work out a plan? Give me a, you know, a bunch of options for my review and there you go. say the the self-deploying nature of this technology feels to me in in my practical use like it's just hit a level that mostly solves these issues if you're willing to if you're aware that it can and if you're willing to like give it a try. I still I I guess I it's so confusing but I guess I feel like we might might have just hit a threshold where re deployment might really accelerate. I I would be it'll be interesting to see what of course you know we're probably not going to see what happens in terms of what would happen to Frontier Model Company revenue if they stopped releasing new models. But I do feel like right now the ones that we have have have have crossed such critical thresholds for ease of deployment and ability to help you kind of rework your process where needed that as that dawn on people over the coming months, you might see a dramatic acceleration in use. Because even just the beginning of this year, you know, when I think back to January when I was really getting serious about personal agent setup, it was a grind. You know, there was a lot of mistakes still and just things that it wasn't quite getting right. And I felt like I had to check. I was doing a lot of checking and, you know, getting AI to do some of the checking, but I I really felt like I felt like at that time I had to be the real owner of like how all these processes were being designed. And then was getting a lot of implementation help, but I wasn't I wasn't able to really hand off at a high level conceptually like here's kind of what I want, you know, can you make it happen?
142:35 And now it feels so much more like that that I suspect we may just have you know another one of these cases where it's like people's impressions are a little bit outdated and a little bit of exposure to what it feels like now to ask for improvements or new new workflows. you know it's it's such a game changer that maybe people just need a little bit of exposure. just in the last day, you know, I was like, "Hey, should we upgrade our embeddings on, because I, you know, we got this deep context database and a bunch of content is embedded and that's helpful for search sometimes when keyword search doesn't work, whatever." basically now I'm just like, please go look at all the new embedding models. There was a new one from out of the Gemma family that inspired this question. Please go look at all the embeddings models. read up on them, see how they compare, look at the prices, look at our options, make a benchmark if you know based on our own use cases, and then test all these things and give me a report. And it can even go do the computer use to sign up with new products I haven't even used before.
143:45 >> So I'm I'm literally like that level of a prompt to get that kind of qualitative, you know, that quantitative result back on which to then be like, "Yeah, okay, cool. Let's go this direction." you know that is like a multiple faster than it would have been at the beginning of the year when I felt like I had to be the project manager still. I I I'm guessing when Sonnet or Luna get to the capability level of Astra or Fable, I think that's when that's when things will start to happen because I think the price and latency you know points have not been met yet. I think the capability the capability threshold has been crossed but I think the price and the latency is not there. I find I find it too slow to use Muse for example. Muse was quite fast when they first launched. I tried Muse yesterday. It was too slow. So I think there is a there there is a problem with capacity and the the speed latency and you know other things. I think so the firm I think what's happening inside the frontier labs is that the frontier labs are seeing the full effect with the latency required to make it make it worth their while to run like 20 agents and be able to like work in flow with them and I think outside the labs unless you are in an organization which is willing to spend like 100 grand you know 100 grand a month you know in expenses you're not you're not really seeing that yet and I think we will we will get there but it's going to take like you know 12 months or so by that time I don't know what the capability frontier on the on the largest models is going to be. So >> yeah, I mean obviously speed is quantity has equality all its own. Speed has equality all its own. I don't I kind of honestly though was thinking somewhat the opposite direction this morning. I was feeling like damn these things are getting fast. you know, Opus 55 is is repeatedly returning reports and proposals where I'm like, that was only like two minutes. Like, how did you do that that fast? I think it's fanning out and doing research a little bit more efficiently is maybe one aspect of it. you know, just as we kind of talked about with sub agents, you know, you can spend more tokens to get faster and and sub agents definitely help with that, especially if things are parallelizable like research, but also I'm like just finding the token efficiency to be really quite incredible. I mean, I've I've done a lot of stuff in just the last two days that I checked my usage and I'm like, man, I got to come up with more ideas. I got a long way to go here. you know, it seems like you can really burn a lot of tokens solving truly galaxy brain problems. Although, as you noted earlier, only like three hours for a lot of these math results, that's not that long. The the kind of simple stuff that I'm doing, you know, where, you know, over the last few days, my personal agent console, hey, can you add voice mode? Hey, can you add better search? So when I'm searching through old threads, I've got like different angles on search features that would be like a week, you know, for a person to map out, figure out what tools they want to use, code the whole thing up. they're they're minutes oftentimes. It really is incredible how quickly you can ship this stuff.
147:17 >> So one one note on pricing. so semi analysis did a price comparison. The same $200 claw plan is worth 2 $2,500 to $12,500 in API value. And so they compared Sonnet 5.5, Opus 5.5, and Fable 5.1 if you use the max amounts what what your value that you could get out of them would be. They also checked in on Astra. Astra.
147:52 so OpenAI after the recent OpenAI recently changed the way that they calculate the limits and after their recent changes, the maximum value you can get out is about $2,2900 out of a $200 a month plan. And this is for Soul GPD 6.1 Soul you have it's now like much lower. So you can see that the pricing has changed over time as these firms kind of fight it out. So right now the best dollar value you can get is the $200 a month plan with Claude and using Claude Sonnet 5.5. however I will note that I have had hallucinations with Claude Sonnet 5.5. So I have never I've never stooped below Opus 5.5. But this is this is semi analysis doing doing their analysis on the pricing and the value from each plan. So those of us and and I you know at this point I believe the $200 a month plan is kind of a loss leader for senior executives to implement at their company. Right? It's you you start off using it, it becomes so useful, you get caught up in this AI psychosis and then you roll it out across the rest of the company in the in in the next few months. So that's my guess on why why we have such high values, such high value given for these $200 a month plans right now.
149:24 >> Yeah, that's pretty interesting theory. As Swick said yesterday, we're all kind of token billionaires in the sense that at 200 bucks a month, you get over $10,000 worth of API usage. And it's with Opus 55, four million $4 per million input tokens, $20 per million output tokens. If you call that say $10 blended average, which is probably a little high, but makes the math convenient, then you're looking at a,000 million or a billion tokens that you're, you're able to get for your 200 bucks a month, which is pretty insane.
150:08 You know, it's definitely the kind of thing where it's like you got to re I think Ze put it really well when he said, 55 should make you rethink your ambition. like, yeah, how what what do I need to do to be processing a billion tokens a month? Like everybody you know, I not everybody obviously, but everybody who's, you know, gainfully employed in the United States certainly can like afford that and the limitation is much more often on can you come up with enough good ideas to use your billion tokens. Trying to hold myself to a high standard in doing that, but can't say I always max it out. I mean, I' I've maxed it out from time to time, but more often than not, I'm probably leaving more value on the table than I should. And it's all because I'm failing to come up with ideas to to really make use of a billion tokens.
150:58 >> I I find I am waiting for an agent coordinator. something that I can I can put all of my thoughts into and then it manages both the all of the various threads on both claude and chatgpt and news and all of these together and I think the firms have so far kind of I I think it's not something that they focused on. I've managed to get Claude to do it by telling Claude that it has access to the Codex command line and gro command line and tell it like this is the this is the way you can access Codex threads because CEX has a particular threading mechanism that that they store the threads in. So you can kind of tell it but it's a it's obviously very hacky and a little bit too tedious for anyone else to do. So I think once that is there and and this may require a device this is this is where I think the openi device is going to come in handy. I will also note that Apple is starting a new round of privacy and security features. they're going to basically to cordon off your private material from agents and to force agents to force you to actually consciously and positively aced to the agents having access to your data and so I think that's going to be rolling out in the next couple of months and so it's basically locking down your laptop and locking down your sandboxing yourself against the agents and that will mean that you know without explicit and positive and affirmative and frequent you know affirmations that you want the agents to have access to this data they won't. So I think I think that that devices are necessary. So >> yeah that's going to be interesting. I mean, it would probably be wise to use those tools to make sure or you you know, respect some of those safeguards, but as Evan said, you know, how many people are just going to turn them right off and just be like just have at it. I can't be bothered with all this stuff. I definitely feel a bit of that pull. Nothing more frustrating than, you know, coming back hours later and being like, "Oh, you got stuck on asking me to approve this one stupid command that I wasn't really even going to read anyway." I mean, that's the thing that's like really tough about it, you know, behaviorally when I look at myself, I'm like, what percentage of these things am I actually really even evaluating anymore versus just kind of rubber stamping or, you know, what's the old the old Simpsons thing where he sets up the birdie to press enter on every on every prompt until finally the nuclear power plant has a meltdown.
153:45 you know, hopefully we're not headed for that meltdown, but I do feel myself as kind of the the birdie that just keeps pecking the enter key in response to a lot of these prompts for approval. That is definitely, I think, a very unsolved problem. How do we actually get people to pay attention when they should be paying attention? it's very delicate because if you obviously you know I'm living proof that if you just ask for permission too much then people just kind of get numb to it and just blindly approve everything.
154:17 >> well what how we've traditionally done it is regulation. How we've traditionally done it is not getting people to pay attention but forcing the companies that deal with those people to you know provide things in a easily understandable manner and if it's too complex to step in with regulation to decide how the framework should work. So for example no one reads their insurance contracts. The insurance contracts are basically decided by the state and and the state will intervene in laws and the state keeps the insurers kind of you know well capitalized changes the rules from time to time. you cannot go to an insurance company and say as a as a small as a small insurance buyer you can't go and say like hey let me negotiate the insurance contract with you why don't you you have this AI which is so smart why don't you actually price every single clause there and then why don't we agree on a different pricing for me right like we don't do that I I wonder maybe if that's the future maybe you can do that but typically because as As a consumer, you don't have a lot of power against a large corporation. The state steps in, the state decides what the contract looks like, and then people don't have to care. I like I'm a finance professional. I don't know what is in a regular mortgage. Like, I have not read through all the laws that, you know, determine how the mortgage that you know, Fanny May or Freddy M would would buy. I I I do not know. I I don't really care either. You just look at the interest rate and you kind of go with it, right? So I think I think those are things that that that's what I think regulation kind of does. It makes it's going to narrow what consumers see to a easily understandable thing that you can ascly without too much homework. and it's a responsibility of the corporation to either provide you that or for the industry to come up with standards around that or for you know the state or the nation or or the country to kind of federal regulation to kind of like this is what is going to be offered to consumers and this is what is going to be easily understandable and that's how you're going to do it. So, I think I think those things will will happen and will have to happen. So, >> well, meanwhile, we've got Haiku 5.5 was just released while we've been live as well. So, the cost curve continues to come down. It is, you know, pretty good and awfully cheap is the the kind of first impression that I've taken away from their posts. 10 cents per million input tokens. and only 1 cent per million input token cache read. So >> I I I I I will note that I have I have some queries running to >> Fable and you know on the API for the show and I will note that anthropic APIs are always slower than I would like them to be. So latency latency is still I think an issue for the firm and I think that's an issue around the capacity that they have.
157:26 I would still be reluctant to rely on anthropic in production like it's it's it's in production live like I I I wouldn't rely on them. OpenAI still has the the API team has done a much better job and things are just faster, more snappy and like you you can tell. Anthropic is still I think trying to they still need to buy more capacity. They they they're they're going to they're going to match Google by the end of the year I think or or by the end of next year. They should be matching Google by the end of next year.
157:56 So they're they're they're on the ramp right now, but it's also you know $5 billion of revenue and $250 billion of commitments for next year. Pretty soon we're talking real compute. >> Yeah, pretty soon we're talking real money. A billion here, billionaire. >> so >> anything else for today? >> No, I think we have we we had a we had an excellent show. We had you know, Evan and Edward from Merkor.
158:27 I I I I see I see a lot of new new new offers for data coming in soon to Merkore. I think I wonder how long this this this round of data data collection is going to last. So we'll be back next week. we are actually going to try and do a Monday next week and and also one day later and yeah I I think I think this this week is going to be a basically a digestion week for people for because it's just just too many things happening I think in the last couple of days. So every everyone has to recalibrate. and I think, you know, we had 200 you know, 20% of the top 4,000 problems getting solved this week. I think if you wait a couple more weeks, we might see another 20 or 30%. Things are not going to slow down.
159:17 So, >> you want to play some accelerating music for us on the way out? >> what shall what should we play? Do you do you want to do you want to hear the let's see. >> do you have a title for your song you played earlier? >> No, I don't I don't have a title yet. It's still It's still labeled. Pre preview Take Nine. Take off. Take off. Take off. version 9. let me >> How about want to learn?
159:50 >> yeah. The models they just want to learn. So, >> models they just want to learn. >> Just before open AI started, I met Ilia. One of the first things he said to me was, "Look, the models, they just want to learn." >> And my kids will never grow up in a world where they were smarter than computers. >> We will have the first time something that is smarter than the smartest human.
160:21 Is AI going to kill us? >> The models, they just want to learn. The models, they just want to learn. They'll do it. They'll do it. We have to slow down. We're going to be on the exponential portion of the S code for a long time.
160:55 >> It's pretty crazy. >> It's just insane. It may fake my books. Not AI, it's SI slop. The models, they just want to learn. The models, they just want to learn. They'll do it. They'll do it. The models, they just want to learn. The models, they just want to learn. They'll do it. They'll do it.
161:26 This is the most dangerous thing that humans have ever gone near. We have to slow down. >> You're not talking to somebody who woke up a loser. Woke up a loser. Woke up a loser. Woke up a loser.
161:57 woke up a loser. Woke up a loser. The models, they just want to lure them. The models, they just want to lure them. They'll do it. They'll do it. The models, they just want to lure them. The models, they just want to lure them. They'll do it. They'll do it. We need to slow this down. >> We have to slow down.
162:28 >> Researchers are continually being surprised at the rate of progress. >> That's sometimes called the hard takeoff scenario where maybe they're extremely fast at doing that. >> Things are moving really fast. >> So, they're immortal. We've actually solved the problem of immortality, but it's only for digital things. Succession to digital intelligence or augmented humans is inevitable.
162:59 >> And the day will come when AI will do all of our all the things. Not just some of them, all of them. We have to slow down. We have to slow down. We have to slow down. We have to slow down. We have to slow down. We have to slow down. Slow down. Slow down. Slow down. Slow down. Slow down. Slow down. >> It's insane the pace. And there's no reason to be this fast. The models, they just want to learn.
163:23 >> We have to slow down. >> The models, they just want to learn. >> We have to slow down. >> The models, they just want to learn. >> We have to slow down. >> The models, they just want to learn. >> We have to slow down. >> Government has to be regulating us. >> Bad actors going to get caught. >> Some creative jobs maybe will go away. but maybe they shouldn't have been there in the first place.
163:50 There will come a point where no job is needed. >> The models, they just want to learn. >> They're able to self-improve, maybe code themselves, future versions of themselves. So at what would be a a slight lead, let's say, you know, a few days could be could suddenly become a chasm. >> The models, they just want to learn. >> Hard takeoff. Some people say it's in three, some people say it's in five, 10. Numbers have been thrown around.
164:20 You talk about 2030 like I don't know what the world looks like in 2030. Hall take off. >> You have a brain. The brain is a biological computer. >> Should we consider them humanity or different from humanity the models they just want world >> there's not going to be a clear distinction between human and machine. The bottom line is we are one human machine civilization. This technology has already expanded who we are and it's going to go into high gear when we get to the steep part of the exponential.
Summary
- The pace of AI development is accelerating, raising concerns about its implications for society.
- Generative models are becoming increasingly capable, with potential applications across various sectors.
- There is a call for regulatory measures to manage the risks associated with AI technologies.
- The importance of data quality and the role of human expertise in training AI systems are emphasized.
- The conversation touches on the challenges of integrating AI into traditional industries, particularly in terms of process redesign.
- The need for effective evaluation metrics to assess AI performance in real-world scenarios is highlighted.
- The speakers discuss the potential for AI to enhance human capabilities rather than replace them, stressing the importance of collaboration between humans and AI.
- There is a recognition of the ethical considerations surrounding AI deployment, particularly regarding privacy and data security.
Questions Answered
Why is there a need to slow down AI development?
The rapid progress of AI models raises concerns about their potential dangers. Experts emphasize the need to slow down development to ensure safety and ethical considerations.
What recent advancements have been made in AI's ability to solve mathematical problems?
OpenAI has recently solved 722 out of 4,000 hard math problems, showcasing significant progress in AI's mathematical capabilities.
What is the importance of policy in the context of AI advancements?
Policy responses are crucial to manage the risks associated with rapid AI development and ensure that ethical considerations are addressed.
How can AI be aligned with public good?
AI development should focus on public good rather than solely profit, emphasizing the importance of ethical considerations and societal impact.
What are the implications of agent identity in AI and cybersecurity?
Understanding agent identity is crucial for cybersecurity, especially in the context of AI systems interacting with each other.
How is data being utilized to improve AI models?
Data is increasingly being used to benchmark and enhance AI model performance, with a focus on economic value in various domains.
What are the current trends in AI training methods?
There is a shift towards using reinforcement learning (RL) and supervised fine-tuning (SFT) to enhance model training while minimizing disruption.
In what domains has AI surpassed human performance?
AI has reached superhuman performance in domains like chess and Go, indicating a shift in reliance from human expertise to machine capabilities.
What are the economic implications of AI service pricing?
The pricing strategies for AI services reflect their perceived value and potential for widespread adoption across industries.