Raising an Agent - Stop Boxing In Your Agent (Season 2, Episode 5)

0:00 Was there anything too controversial that you did not include? Where did you hold back? >> I do think like one hot take that I didn't put in there is seeing how conservative and skeptical and thinking in the box most engineers have turned out to be over the last year. It boggles my mind. Like I I assumed we're all like interested in technology and doing being good as engineers, but then over the last year it's like guys like don't you know anything about the history of the technology you're working with? You know, they they say stuff like oh it can't do this. And I'm like you don't know like how Java was perceived. You don't know how JavaScript was perceived.

0:38 You don't know how Pascal was received. Like have you never read a book about programming? You know like we've had this for the last 40 years where a new thing comes out. Personal computer. Oh that's nothing. Like that's a cheap thing. Every technology had like these voices. I'm looking around. I'm like, "Guys, have you not read anything about the history of programming? You sound just like somebody in 2002 saying garbage collected languages. This will never fly." You sound like somebody in 2010 saying JavaScript. That's not a serious language. Like it's it's crazy to me. And the other thing is that do you not realize like what the function of software is? Like you have to realize it's not the stuff that you do in your free time in open source. You have to realize that in a business it serves a purpose. Like if so if if if somebody can get away without hiring software engineers, they would do it.

1:23 Like it's you everybody got to realize you're a means to an end. Like you you know people are not realizing just how much will change >> like planning is dead or it's going to change 99.99%. Making slides, documents, a human emailing another human. Most of the reasons why a human emails another human. Everything about how a company works will change.

1:55 Welcome to Raising an Agent episode 5 season 2. And we're here because we feel it in our bones that we have to record this episode because it's hot take Thursday. We just called it this because there's so many things we want to say and there's so much going on. We we couldn't wait, right? It's in the middle of the night for you, Quinn. I guess like you're night person, but like still like 2:00 a.m. hot take Thursday. Yeah, I would not miss it.

2:23 >> Strike the iron while it's hot. You know, hit the hot takes while they're still hot. So, just to set the scene, yesterday evening, I was at the birthday party. I didn't expect to talk about AI at all. like in the middle of nowhere. Didn't expect anybody to talk about agents. And then somebody actually brought up agents at the dinner table. And I got to talk to a relative of mine who works in a German consultancy, you know, small city German consultancy.

2:49 They have 100 engineers, so maybe not that small. And they do it and software consulting for companies that outsource their software. And he told me, "Yeah, they've been using agents." And they started out last year with like a sub. Everybody got a $200 a month sub with all of the tokens that includes and then leadership said that's too expensive or I think I guess maybe they got priced out right like most companies like the Anthropic or OpenAI they say if you have more than 100 users you cannot use the subs or something like they had to switch to API pricing >> around that >> yeah something like this and they said it's too expensive we're not going to do this so then they started I think adding a model router you know where they kind like, oh, some things we route to cheaper models. Like, not the engineers decide which model they use, but somebody, I don't know, like, >> yeah, if the salespeople took people out to an expensive dinner the night before, then the engineers are going to be using Quinn, right?

3:51 >> Yeah. Something like this. So, that led to every engineer having a token budget of, I guess it was euros, €50 a week. Okay. which is at API prices that's not a lot and and and my relative here he ran through this in in a day. He he was like I need more you know and I'm like a day that's already like you're doing like that's crazy that you managed to do it in only a day or like you know couldn't use it a whole day. So then he asked for more. He got more and then they were like, "Nope, like that's too much. Like $100 a week, that would be too much or sorry, euros, that would be too much. Too expensive. We can't do this." And now the latest is they switched the whole engineering org or all the engineers over to use Quen. And I couldn't believe it. And he he's like it's horrible. Like it's dog slow. and and and that guy is not like he's not as AI pill or token pled as we are but he was like it's dog slow and you also don't save any time nor money because if it fails the last 10% you have to redo it again or you got to figure out where it failed like where those 10% are right like if it gives you 500 lines of code but you're not sure whether it's 100% correct or 90% correct then you got to spend time and figure out where the mistakes are so this whole idea of let's just use a cheaper model as you've seen over and over again, it bumps into the reality of well, if it's wrong 90 or 10% of the time, how do you know when the 10% is? And isn't it better to pay a little bit more so you're sure that the frontier model doesn't make those mistakes?

5:32 >> Yeah. And Quinn is not a bad model. I mean it's a great model but if you are paying very expensive people in a competitive market economy >> and your competitors are using something better than Quinn then it doesn't matter whether it's in you know it's a good model or not if it is if you're at a disadvantage then that's all that matters. >> Yeah. I I couldn't believe so he was saying it's so bad and then it's useless like he uses like he has a sub at home.

6:03 He's like, "It's this is so much better at work. It's useless basically, but they're saving money." And I couldn't believe it. Like, I I I guess I get it if they're like a German consultancy. Maybe they don't have the profits to afford AI, but like what what's the plan here? like because you your engineers they what I tried to tell him yesterday was if you're using these models from 6 months ago or a year ago and I guess you could say when it's like a year ago this is a completely different category of of tool like what you're doing is you're still stuck in this like oh the agent writes like this much code and then do this and I'm over here using an Astra and a Fable and now an Opus 55 and it does everything right like it it ships to production it monitors the logs, it debugs stuff, blah blah blah. Like, it's a completely different thing. And to me, it's this it's a category change. And if you're not on the frontier, it it feels so strange that as the leadership in the company, they're like, "What are you doing?" Like, you're you're basically saying, "Oh, the internet is here, you know, like and apparently the leader of this company said like, yeah, AI is going to be the next big thing." So he's aware of what a revolutionist is, but then you're giving everybody like dialup modems while you could have fiber, you know, and it's like, yeah, like internet is good, but I'm going to give you dialup modems even though fiber is already here, you know, in this parallel universe where you have both. And and and what my point is, say Gwen is the dialup modem and like the latest frontier model is fiber. Yes, it's the same thing, right? It connects you to the internet. But think about how you use the internet. Like with fiber, you stream stuff. You open YouTube. You send files around. You upload stuff. You don't compress your videos before you upload them to Google Drive. You know, like you have Google Drive running all the time. If you have a dialup modem, you use it completely differently. Yeah.

7:58 You might disable automatic image loading in your browser, you know, like. And that I think is the difference between last year's models and today's. It's a different category of of mods. like you just do completely different things. >> Yeah. And it's not just that the individuals use it differently. It's that it makes it so the company can pursue completely different business models. >> And look, I think we both sympathize with people at companies where the companies don't know how they are going to afford the necessary token spend for all of the people that need tokens. That is a very tough thing. In the same way back in the 1800s during industrialization, there were a lot of people that were working in their home lowcale production, you know, they would be just doing the loom at home basically and they would see these people going off to the factories and these people working in their home had no way to raise the capital to go build a factory.

8:55 And I think we all sympathize for them. But in a way the universe it doesn't matter whether that person has a sympathetic reason. The reality is the world is changing and things are being disrupted and you cannot just plug AI in. You have to think about how it's going to change everything. >> Yeah. >> And that's a scary thing. And so, you know, for students, I think if you're someone who is not able to experience what it's like to build without, you know, real token constraints, if you're someone who's working at a company where they're putting really intense token constraints on you, then try to find your way to somewhere where you are able to build more freely.

9:43 >> Yeah. >> I I know it's easier said than done. I know that's hard for a lot of people, especially people that don't have local jobs where they can go and and companies where they're willing to pay for a lot of tokens, but I think if you're listening to us, you are I I think you're pretty smart and there are companies that people do not have to care about token spend. And I don't mean people are doing dumb stuff with it. I mean people have skin in the game and they're conscientious. And you know, for us on AMP, our team is 20 people. We have no kind of token limits.

10:15 when we hit the limits of our subs that we plugged into AMP, then we just switch them off and we go under token spending and there is just a a freedom to build in this crazy time and it feels like that's what's needed to to get ahead and to figure out where this all going. >> Yeah. And I think I you know I wrote a tweet on Saturday like five six days ago and it blew up and it was about how I see the future of software development and most if you've listened to episodes of this podcast like most of this wouldn't be new to you but I put in I said some people for the next few years will be priced out of developing Frontier software or like the new kind of software and people got mad and >> you don't want that to happen. You're not saying it's good. Exactly. I said I think this is going to happen. And I'm seeing it now. Like I'm seeing how people live in different worlds. Like I talked to people who use a Quen at their German company versus you go to somebody with unlimited tokens in San Francisco.

11:15 It's a completely different thing. And the reality is >> actually I I want to say something companies in San Francisco there's a lot of them actually where they're putting on intense token constraints as well because they have more engineers per capita than a lot of other companies and they're at the same time being threatened if they're a software company traditional software company. I love that we can say traditional software company now and nobody blinks then they're under a lot of pressure. So actually it's it's a lot of these new waves of companies that have started in the last couple years and then it's also companies like from the US Midwest where they have people that are wearing a lot of different hats, really smart people.

11:55 They didn't have a big software engineering team, but now they have a few people that are spending a bunch of tokens. Yeah. A lot of San Francisco companies are not the ones that let you spend freely. >> Yeah. And look, like like I said, I don't want this to happen. And then people get mad and saying like, "Whoa, whoa, well, we have open models, we have open source models and all of this." And yeah, that's true. But man, like look at the 80s or 90s even. I don't I don't have the numbers, but basically people said like if you were able to buy like an Apple in the 80s for your kid or something that's equivalent to, I don't know, $10,000 today or like you know like it's it's the price of a small car or something like it was expensive. So people had been in the early days of personal computers they were priced out of con you know participating in like oh I could develop software at home and all of this like most of the time if you had access to a personal computer at home in the 80s and 90s your parents were well off you know if you were a kid and people in the eastern block they didn't have access to the new stuff it was a whole different culture that developed there with their own programming languages and whatnot. So it's a thing that happens where like different industries or tranches of the same industry develop because of different access to capital and and hardware and whatnot. And it's two different worlds if you use like these open weight mods and or if you have access to the Frontier model. And to tie it back to what you just said, people say, well, you know, the company got to save money and blah blah blah, but I think it you're playing with fire. Like the company might save the money now. And yes, you might survive for the next 5 years doing what you've been doing. And it's not like, you know, when people now say, "Oh, where's all the AI stuff and where's this?" It's not like people will shoot up right now and disrupt you and build the same thing that you've done.

13:40 It will be that you will slowly bleed out like you will lose growth rates. You will lose new customers and whatnot because others are faster than you. So if you're the leader, you got to think like what hard decisions do I have to make now? And I even said to him, my friend yesterday, I said like are you guys like you know cutting people like firing people like what's the strategy? And he's like, "Yeah, we stopped hiring." So they're in a stasis mode and now they're giving out these cheap things. And I said, like, as tough as it sounds, like for a lot of companies, the playbook is maybe something like disrupt yourself. like you need to create small teams and give them the most leverage, you know, which is what we're doing with AMPLABs, which is have been tremendously successful, which is exactly this, like yeah, you you you can't give 30,000 people in company frontier models unlimited, but what if you create an elite team and give them unlimited tokens >> and not just for cost? If you were to make the agent that you could roll out to 30,000 people, everyone at a company, then it would have to be watered down.

14:42 That's just the reality. So, okay, another way to look at it is let's say that you are a factory before like robots and automation comes along and you've got all these factory workers and then all of a sudden robots start to be invented and you have some people they go and bring in their robots from home and they help out and you know that's pretty cool. And then you know the factory says all right each person has a 50 euro per week robot budget. you can bring in a robot for only 50 euros a week and you got to make do. And so people come in and they're working on the production line and their robots are helping and then oh man it's the seventh of the month and their robot budget is up so you know they send the robots home. It's like that does not make any sense. A company needs to think high level about how it's going to use this technology. It can't just abdicate its responsibility to to figure out its production system to employees. It makes absolutely no sense.

15:41 >> Yeah, it I mean I guess we could talk for two hours about this, right? Like we we got more hot takes, but like I'll maybe I'll summarize it like this. Maybe maybe maybe how about this. Like AI and these frontier models is not just like a better IDE that you give your engineers. Like it's not just a thing, a tool that you hand to these engineers. Like you really got to rethink from first principles. Why do you have an engineering department? What are the goals of those engineers? Like, and how how can you reinvent this with AI and rethink the complete or structure, the processes, everything? Otherwise, I think you're just missing out on a ton of benefits. And yes, you might be fine for the next few years, but it's not like somebody knocks at your door and says like game over because the reality is nobody will knock at your door when the game is over. You know, like the absence of somebody knocking at your door means that the game is over. It's hard to hard to hard to notice.

16:38 >> So, >> all right. So, in keeping with the theme of this is like a tsunami, it's a wave and you know it's hitting everyone. Talk about how the agent is expanding from having done coding for you. >> Yeah. >> To doing more. What does that look like for you now? So I'll start with the hottech version of this which is what our very own Alex Camper I think he tweeted it by now but he said it internally of >> he is such a wise man.

17:05 >> We have an awesome team but yeah he is just wisdom. >> Yeah. Yeah. He he had like three or four of these Slack posts where I keep thinking about them for a week and then I'm like I wish it was my you know I wish I could tweet out. But what he said was based on an observation of I think it was like some large corporation he had insight to or whatever where they shoehorn the agent basically into the text editor and say well you can write code but before you access anything else you need to ask for permission and do this and blah blah blah. And I think he he was working with them he had to get like a new deployment setup or something like this and he had to get permission in like one system to get access to that system. we had to ask for more permission to get access to this. He had to find the right department and it was like a Kafka-esque adventure of like getting a new deployment set up and he said and then people wonder why AI doesn't increase the GDP you know the bottom line being if you now have like these 10x multipliers but what are you going to multiply if you're boxed in and you need to ask for permission all the time and I think that's the same thing with agents. I think most people right now use agents still boxed into the idea of it's the thing that writes code and I review it and then I push it up and it's it's it replaced my text editor. I think that's not aiming high enough. I just had this conversation on Twitter where I said based on these cheap models like if you use a Frontier model like the newest ones an as a Fable Opus 55 you will realize you can aim much much higher. So previously the agents went fully vertical in the stack. They started with the front end stuff, the little scripts, then they did backend changes, database, whatever. And they're fully in your stack. They they write your code in codebase. But now I think it's time to go horizontal meaning the whole life cycle of you know software development life cycle. They can prototype features.

19:00 They can pick prototypes. They can document them. They can write the features. They can ship them through production. They should monitor production. They should debug production. Should keep an eye on logs. get bug reports, fix bug reports, talk to customers. Like this is everything we do. Like on Monday, I found a bug for example in AMP. The agent found it incident like by accident in the logs and then we investigated because it has read only access to our production log.

19:25 So it can see what's happening in production. No sensitive information, right? But it found the these errors. It was like, okay, let's investigate this. So then we found a fix and they said like, huh, let's roll this out and let's keep an eye on it. So we I rol pushed it up. Then it monitored the rollouts in our CI. It has access to our can see our CI. It can see our deployment so it can know when a version is live. Then it uses automations to remind itself every hour to check on how it's doing. And then it would monitor for last 5 days.

19:56 It would monitor production and send me reports like after every deployment. Oh, error rate went down. Oh, this is good. Oh, here's another thing that pop up. But it's unrelated. And it's it's wild. like it's end to end the whole thing and now other stuff started popping up. I tagged you in a thread but it was unrelated and then it spawned another agent in another orb to investigate this. So it's fully integrated into how we develop software like from local dev or orb dev to production. And >> I I I think if you're not doing this and you still think if you still try to box it in, it's just you you're losing so much. you. Yeah.

20:35 >> And like it really feels like sometimes we're driving and sometimes we hand over the wheel to the agent and I would say like more of the hard decisions we still make. We are still the ultimate decision makers, the designers, but there's a lot of really hard things where we fully delegate to the agent. And you know, one really concrete thing, two weeks ago, even a week and a half ago, I was using the ship button in AMP where you hold it down, it has that nice animation all the time. And that that was awesome. You know, that would just ship it to main.

21:10 And now I'm finding that not only do I do more stuff that the agent helps before I actually start having it do any code, but it's almost always, okay, let's deploy this a little bit. Let's see all the other users that were affected by this. Let's add some more logging. Let's let it run for an hour after deployment and then tell me was it accurate in its prediction and then you know make this graphana dashboard monitor that and once it's been rolled out check the logs every 15 minutes for an hour send me a message if there's any issues otherwise archive it or email the users that were affected give them all $10 in all these things and it's kind of absurd. I actually am really leaning on dictation because there's no way that I would want to type all that stuff out and it's clearly a gap in the product where we need to figure out how can we take some of these patterns of things where you don't roll them out and make it so the agent can just do more and then you know we've got this bug reporting system and someone was asking us and I was in Australia with some customers and and meeting a bunch of people if you came out to our meetup then it's great to meet you. and we showed off our bug system which we just built. I mean, you built most of it, Thorston, and it will take bugs from the product. It'll gather diagnostics, anything that people want to attach.

22:31 It'll show up in our Slack, and we've got a I'm feeling pucky button and I'm feeling ultra pucky button. And we click, and that kicks off a thread, and that is the start of about half of our work right now. And it has created this virtuous cycle. So, you know, if you picture, you know, one of those, you know, mazes like a corn maze, you're looking down. I don't know. Do you have corn mazes in Germany?

22:56 >> We have mazes, but it's not corn. That's >> Okay. >> Yeah. Yeah. Yeah. >> It's, you know, sparle asparagus or something, >> you know. Well, I'm from the Midwest and, you know, we got corn mazes even here in California. And it's like, you know, the agent starts at the middle and you want to clear out all the walls. You want to let it just grow in whatever direction it's going to be able to help you. And we are able to do that on our team and so many other teams that we work with are able to do that. But if the agent starts to, you know, help over here more in the beginning of the process and then that's what you're using GitHub issues for or linear for, well, it's not going to be able to grow.

23:35 you know, it's probably going to grow around that wall and then smother that wall and you're going to end up somehow you've built your system on GitHub issues or Jira and nobody actually uses it anymore. You don't need it anymore. The agent is driving it and you should have just gotten rid of it and it's probably going to be subsumed by the agent anyway, but it's going to slow you down. If you've got code review on the other side once you're done with the work, if you've got all these other processes meant for the wrong thing, you know, code review in so many companies it's a farce now. Nobody can review the volume of code >> and they're not even the best at at reviewing it. Yeah. And you know it's clearly it it's a farce. It's just slowing you down from actually benefiting from the velocity improvements. And if you are you know let's stipulate to you do take on some more risk relative to the perfect ideal code review process where the smartest human John Carmarmac is reviewing it a thousand times over for a thousand years. You do take on some more risk.

24:31 So, what can you do to balance out that risk you're taking on? You can move fast. You can have people get skin in the game. You can keep people in flow. You can have the agent monitor things after. And if it hits this wall where you got to get out of the agent, you know, you got to get out of your car and you got to walk over this gap and then get back in a different car, that is not how the world and technological progress wants you to go. And yeah, you're gonna be hitting your head against the wall.

24:59 You already are. >> Yeah. It's like when people say, "Well, I wish I could do this, but we don't have a mono repo. We have 10 different repos." And then it's like, "Then get a mono repo." You know, like this technology is so powerful that this is the time where do you have to rethink a lot of these assumptions and rules that you've previously put into place. You know, no dev tool can access this or no def like maybe now's the time to rethink this. Maybe now's the time where you got to give it access to this because you're boxing it in. Let's say that let's talk about what you would do if you were a smart engineer who is at a company and you feel like your company just doesn't get it. Like how would you speak up?

25:41 >> What can you do? >> I don't know. I would I would complain a lot. Like you could like there's some remind me when you kind of you have to break the rules a little bit. I think like that's that would be my approach. Like I don't think you can >> there's no German word for someone who has to break the rules. >> Yeah, there's no German. >> That's the only word that does not exist in German, right?

26:05 >> You can't even compound it. The police the police would show up. you cannot do this on paper. Like I think you have to show this to people. It's the same with like everybody else in that trying to adopt AI. Like you have to actually try this. Maybe there's a safe way to do it, right? Like for example, friend of mine, he works in a company that does small medium businesses, sets up IT infrastructure for them. And he's like, "So, would you really run agents on like these customers machines, you know, to set them up?" And I was like, "Yeah, I would." And he's like, " nobody dares to do it." And then I was like, well, I can't tell you to do something, you know, you shouldn't be doing, but what I would do is I would maybe replicate like a customer setup one to one, you know, maybe with anonymized data or whatever.

26:53 And then say, look, this is how you do it like with the normal setup, and this is how I would do it with agents. You know, I think you can do this in a lot of companies where it's like, hey, if we give the agent access to this and this, then it can do all of these things and you just demonstrate that it's faster. But I'm not a big company politics guy. Like I would I would probably try and sneak around and ask for forgiveness and then say look like it's 10 times faster.

27:18 But this is another hot take just to get this in. Imagine working at a large company right now where it's all day every day AI generated messages in Slack and Teams and notion and PRDS. It's it it would be insane like slide presentations all of it. So I don't know like I have zero. So funny because it's like play acting where >> the idea of most slide presentations it doesn't make sense anymore >> and I mean I I I saw this I've had some customers who are coming to me for help and they have a presentation with a timeline and very formal plan for how things should go and they say hey can't this be faster with AI and the answer is yes, but in a way it can be so much faster that it's not about going back and making a better estimate. Even it's the way that you should be thinking about planning is if you're building an app, you should be using the app. You should get it rolled out as fast as possible and show that you can iterate.

28:26 And partly that's because that's what AI does. And partly it's because like look, we're humans. There's nothing that gets people to believe like they get the app on their phone, they dog food it, they have feedback, they go to sleep the next day, it's already better and all of that. And that is absolutely possible. And most people are not at companies where they can even conceive of that being possible, of a product getting meaningfully better overnight. That, you know, to your point, what would you do if you were at a big company? It's actually showing what this can do that helps the business. Going back to your example when you were at the the dinner last night, like there's just a lot of people at different parts of the spectrum. There's people that have played with this. I think there's people probably that say they have more experience, but they actually haven't used the latest things cuz they don't want to be embarrassed. If you actually sit down with them and show them what is possible, that goes so far. Like what I found is in my, you know, kind of personal life with with friends and family, I've got my parents, my aunts, actually showing them what they can build in five minutes. And also, you know, my kids obviously they they have no conception. I mean, they're, you know, seven years and and five and three, but actually showing people what this can do. Most people just do not know even if they talk about AI, even if they're they have a title of like AI strategist, they just do not know what it is capable of. Yeah, they I mean I've said this before here, you know, like I think the biggest reason why not more of the world is going crazy over AI is that so far only programmers understand what's going on when you see an agent run in the term or you know run commands like we as developers we know it's running programs like it's it's using the computer when it executes bash and shell commands like we see this and we're like holy it uses ffmpeg Now it SCPs it over to the other computer now it la you know like we know what's going on but to the normie it looks like gibberish like oh it's programming you know whatever like that's not my world I use a computer use is the first thing where they go oh wow because you can see the cursor move around right and and it clicks and it uses the computer like like I do or we do just like it did for programmers where it used the bash like we do except 10 times faster with more oneliners and crazy without any typos or whatsoever. So >> yeah, >> I think if you could have computer use let's say 500 times faster or the speed of which you can use shell commands, right? Then I think people would lose their mind because then it's going through their outlook and clicking through the emails and adding them to Excel spreadsheets like 10 times faster than they could do. But until they can have this impression that this is what it's doing, I don't think I don't think they get it yet. You know, like I I think you can show them some stuff, but it's not legible to them. It's not something they can grasp. And >> but I think that you're going to see, you know, like the the Chinese weeklong holiday is coming up pretty soon and you have people migrate all throughout the country and and show people what this actually looks like. You have in the US Thanksgiving, families coming together, Christmas. And I think that a big reason why it really took off for coding last year was because people shared it with their family, they shared it with other people, and they also had time to go play with this. and think that things like personal agents doing this for your life. I think that it's going to take off in a huge way over the next few months. So, that's going to help a lot more people see what it is capable of.

31:56 >> Yeah. All right. Next hot take. Don't wait for your agent. You're wasting time. Let them wait for you. What I mean with this is if you are still using agents and you're watching them work, say you have four of them and you're looking at them and waiting for them. If you're even interested in like, well, I'm interested in inference speed, but if you're like, oh, well, you know, why does why is this harness faster with the first token, whatnot, there's like some edge cases, but in majority of cases, if you're interested in how fast it can produce like the first thing, like you're wasting time. I think I've said this the last time, right? And now it's even clearer to me when I talk with people about verification loops where they're like, "How do you do like how do you, you know, we we talk about this a bunch like we say to the agent, give me irre irrefutable proof that this works, right? And then it goes off and it does its thing." And I don't wait for them. I don't watch them. They're in orbs. They're running somewhere else.

32:51 They're not on my machine. They're not blocking anything. I just think like it's so much better. And I'm never waiting for these agents like like I'm doing other stuff. Like if I have five or six running, then I focus myself on one other thing. Like I write something or read up on something or review other stuff or whatever it is, but I'm not watching a single agent. So >> how many do you usually have running at any given time?

33:14 >> Right now I'm looking at it two, I think. But I think not like you probably have five times that, right? But I think I have peak like if you would graph it, I think it's always one. Peak is like five or six, you know? Yesterday I had 22 running because I asked it to eval something and it started 20 orbs to test the change but I'm not looking at them. >> I'm not I want to go deep into that. We do this a lot where it's an eval. And you might think an eval is something where you have to have a PhD in computer science and be an AI researcher. No.

33:49 Like for us, you know, an eval might be like very concrete example for having AMP manage its own settings. It used to be that it would use the CLI in orbs and if you're using Puck, it would use a server tool. Well, that was different and I standardized it and I made it so both had a tool. And then I just came up with I had the agent come up with a bunch of use cases. What are the kinds of things that people ask AMP to do to like change its own settings or make a new project or add an open router connection? I went through our bug reports where people ran into issues to help seed it. And >> when you say when you say you went through it, you told it to go through it. Yeah. Yeah. Go go go and look through all of the bug reports and analyze it, right?

34:33 >> Yeah. Yeah. >> Yeah. That's right. And come up with a corpus and then try it using the tool descriptions in both orbs and puck and make sure it works. And it it found a bunch of issues and it cleaned a bunch of stuff and and it did all of that. And then I said, "Now run it again." or I recently improved our browser use in orbs. It's not yet released, but I think that browser use in orbs, if you think computer use is cool. Well, imagine not being tied to having it do stuff on your computer. It gets in the way. It can only do one thing. You got to keep your computer open. And browser use and orbs just had it make a bunch of synthetic applications like a shop front something to do like local planning permits and click through them and you know put up annoying cookie banners. I, you know, had it do all that kind of stuff and it does it. And I'm sure that there is a better way of doing eval. It could be even more perfect. But it's like with usability testing, you learn most of the insights from the first three people you talk to. You don't need to talk to 100 people.

35:28 >> Yeah, I I this is going to be live in the next two days. I recorded the tutorial video about this yesterday. So the setup was I have this little web app. It's a Laravel app. It has a login, username, password, but imagine it's oath, two factor off, whatever, like some complicated thing. And I want the agent to be able to go through the application fully and test this with as few steps or chances to go wrong as possible. So I did what we have an app and we documented this in a post called putting an agent into an orb where we said we have our deaf routes. So we have only in dev environments we have like slash_c_dev route where you can say dev/log me in and you I think it's a get request and you send your username and then you're logged in. It's only deaf only and you got the cookie set and you're locked. You can skip the whole flow. So in that video I said to the agent I want this in this application and it build it and then I said and now I want you to document that agents should use this in the agents MD file in the root. And then I said, "And now I want you to spawn three orbs, three agents that know nothing about this change and give them a task with which you can test that they actually use this new route like that they discover it." And it did it like it spawned and it sent them a message where it says like, "I want you to take a screenshot of the admin dashboard or something like this or log in and figure out how many newsletters are published or something like this." It like these models are now so good. Yeah, I think it's because of how they're trained of like evaling each other or or like doing these test scenarios that if you tell them this is the thing I want to test.

37:06 Help me spawn and test other agents. They know how to write instructions. It's really really fascinating to see. >> Well, I mean it's not that hard of a task even. >> It's not that hard, but this was a dumb example, right? Like you can go even harder, but it's it's fascinating. And here's the the surprised me in the video and you can see it in the video how surprised I am that it then spawns even a control group orb where it gives it a different task and says I want you to use the log in form you know and blah blah blah do this so to make sure that every path still works and then you know in this example the agents MD instructions were good but you can do this for everything in your codebase if you want to make it agent friendly as the agent spawn other orbs go through the flow and make sure that it actually picks up on like uses the browser uses the admin thing uses these tools and whatnot. So that's like that's really powerful.

37:53 >> Yeah. And having orbs makes it so much easier to run these little mini experiments, if you don't have a isolated, reproducible, standardized environment where you can run a lot of things in parallel, then there's even if you have a really good work tree setup, there's probably going to be just enough friction. Yeah. >> Do you really want to tank your laptop for the next hour and probably run into weird things with like file descriptors or Postgress or you know something you know or like port conflicts you know >> no you don't you know in the back of your mind >> let's address this like we shipped a lot of I did like I shipped a lot of improvements for runners which are our way to run and remote control AMP agents on a machine that you own as opposed to orbs which are ephemeral remote machines right And people said, "Well, I thought you guys were all about non-local deaf, so why do you have the runner?" And let's address this. I think runners on different machines are non-local, too, you know. so I I have one running on my Raspberry Pi. That's a valid use case. My wife yesterday came in and she was like, "How much electricity did we use? Do you have like the the count?"

39:04 And I was like, I pulled out my phone and I switched in the AMP app. I switched to my private account and I I talked and it said give me the rundown of blah blah blah and it did it. It used my graphana where I tracked it blah blah blah. So I have this running on my Raspberry Pi valid use case and she probably thought they're really cool. She was like wow. >> Yeah. Didn't didn't even comment on it.

39:25 Like it just didn't I don't know. I was kind of let down. I thought you would say something. But anyway, so that's a valid use case obviously. And then the other valid use case is to complement orbs. So we have bunch of customers that use orbs. They love orbs. They say it's orbin time. They sign the emails and say it's orin time. And yet they do have stuff like hardware that they need to test. Like they have machines where they have like you know one specific customer. I don't think we can name name whatever like they have hardware attached to their machines. They have a fixed amount of these machines in their office and they want to be able to run agents on. So their setup is they use a bunch of orbs but then they have a fixed amount of these machines. So what they want to do is run AMP runners on them and then the orb agents can spawn agents on the runner machines right to make use of the hardware or if you need GPUs right we don't have orbs yet with GPUs so you can have like a bunch of GPU machines if you want to test shaders or whatnot you can use them as built machines or Mac OS machines we have one we have the Mil Valley Mac OS built machine that's sitting in your room I guess >> right here >> here we go I'm holding up a M4 MacBook Pro for those of you on audio >> so everybody can access this on the team. By the time this episode come out, I'll have released the feature of shared workers. If not, this is maybe slightly embarrassing, but my point is, >> so okay, this actually ties to the triad of PM design and engineering. And there have been some people out there that say what, you know, why are you doing runners or AMP? You used to be about no model choice and now have a lot of model choice. Yeah, >> like you know why are you doing this?

41:05 And I think that we don't operate where we have a nine-month road map made by sales that then gets filtered to PMS and you know then designers and then engineering. Obviously we don't have that. I think a lot of companies still have the triad of PM design and engineering. I think that when you have a smaller number of people that have more skin in the game that are across more of the life cycle of the the company's like value creation I'm not an MBA like I don't know what that means basically but it's like people that are talking to customers people that feel the demand in the market and people that have a vision and they they go and create that and they champion an idea then you operate in a different way and when it is so much cheaper to build stuff and also And this is crucial when your customers are more open to change.

41:58 Our customers are incredibly open to change. Even Salesforce's customers are probably a lot more open to change than they were like 2 years ago. Then you can try things and and you can take them out and and it's not this whole false start. And so we talk about our team follow your energy because we know that if someone has a vision, if someone is motivated to do something, if they just want to try something, it's not a high bar to go and just try it here.

42:26 >> Yeah. >> And it doesn't need to fit nicely into some clean strategy. We're also all figuring this out as we go. So for those of you who are using AMP, if you give us feedback, if you if you're telling us that that you want to use this thing and you would crawl through Glass to get it like some people are for this little clawed code thing that I shipped, I think it was earlier today, then that's actually a really important signal. And even if it doesn't align with, you know, whatever official strategy that someone might have had 5 years ago, we're going to try it.

42:56 >> Yeah. But you just I I was going to say here's my strategy. I was going to say this like this is my grand strategy. The the what is your grand strategy? >> The grand strategy is everything is orbs, agents everywhere, controllable from anywhere. And I do think runners complement the picture. I don't think it's the future of like running agents. We said this a thousand times. I don't think that's it. But they do complement the picture. They they they are a bridge to making orbs work in your setup. Some people just, you know, they can only use orbs for that stuff. and the runners kind of fill in the blanks and make it work. So that's the big strategy. I think it makes sense. Like it's it's just now that people I I ship like work through support and people have like oh yeah I have like a custom work tree thing. I'm like you know that's already too much. So we got to dial it in. But to your point of like everybody in the team the other thought I had I think I've said this before too is the ultimate customer of AI is an entrepreneur. Like the ultimate user of AI is entrepreneur. What I mean with this is somebody who wants to do something like somebody who who wants to affect change in a certain way and take that in the most abstract sense possible. If you are an engineer who doesn't care about the business goals and just cares about I I mean I'm rehashing conversations that everybody had the last 3 months but if you only care about the writing the code and ship you know whatever like building a nice thing and you don't care why the tickets get handed to you and who's the customers and what their problem is and how you produce value and how the company makes money I think that game is over but if you are interested in that and if you know our customer wants this so let's make this happen then you now have been handed the ultimate tool and I think in some sense that everybody in our team they kind of act like mini founders entrepreneurs in the sense that pick Lewis for example like Lewis I think on Monday or Friday he was like yeah I'm going to own this like I want to make this better and it's not like one of us said go and make this better like he talked to customers he saw the problem he's like we can make this better this is a gap in our product and now he goes and does it and I don't know what the hell he's doing I think he posted an update earlier today that I haven't looked at but like you need somebody like you don't you you want people who who move stuff who push stuff who who manage to push stuff along and then AI is the ultimate tool for this.

45:08 >> So >> yeah, >> and every week we have twohour meetings. We have one I'm based in California, it's 11:30 p.m. on Sunday night and then it's 9:00 a.m. Monday morning for me and you know if you're in Germany or Australia or whatever, different times and that's how we get aligned. But then throughout the week, I mean, we're talking a lot. We talk all the time, nothing's scheduled, but the agent AMP itself, we built AMP to be the thing that helps us stay aligned. And one of the things is it's so easy to see what anyone else is working on. It's so easy to see what was someone's intent behind doing something. And that makes it so easy to feel like you can do anything and you don't step on anyone's toes. You don't have to ask them the mundane questions of, you know, what what did you mean by making this change? That's the benefit of using a product, an agent where everyone on the team is using the same agent and it's built for multiplayer. It means you actually can work this way.

46:06 >> Yeah, there's a lot of that going on. I mean, we just had like some production errors, right? No, this was before. Sorry, I was looking at some other error in the logs that I found by accident. And then I was like, why is this coming up? And then it found the commit that added it. It found the thread that added it. they found out who, you know, wrote the thread, what the motivation was, what they did, and it basically saved me, you know, from reaching out to them on on on Slack or something. So, >> yeah, >> it's coming together.

46:33 >> Yeah. Was there anything too controversial that you did not include in that post last weekend? Where did you hold back? >> This is one thing. Well, I I guess like I could have formulated everything a little bit more extreme even, you know, where I said like, "Oh, the deaf tools won't matter in the future, will be washed away by tokens." I do think like one hot takeake that I didn't put in there is, you know, how most people are sad about like losing the craft and like gh AI, you know, is being shoved down our throats and I I miss typing and all of this. Like one thing that I find myself being really disappointed by and that makes me sad and frustrated is seeing how conservative and skeptical and thinking in the box most engineers have turned out to be over the last year. It boggles my mind. Like I I assumed we're all like interested in technology and doing being good as engineers, but then over the last year it's like guys like don't you know anything about the history of the technology working with you know they they say stuff like oh it can't do this and I'm like you don't know like how Java was perceived you don't know how JavaScript was perceived you don't know how Pascal was received like have you never read a book about programming you know like we've had this for the last 40 years where a new thing comes out personal computer oh that's nothing like That's a cheap thing. The iPhone, you know, like every technology had like these voices. And I'm like I'm looking around. I'm like, "Guys, have you not read anything about the history of programming? You sound just like somebody in 2002 saying garbage collected languages. This will never fly." You sound like somebody in 2010 saying JavaScript, that's not a serious language. Like it's it's crazy to me. And the other thing is that I guess like maybe I'm interested in all of this like the whole business and startup and and finance background with a lot of this stuff but guys do you not realize like what the function of software is like you have to realize it's not the stuff that you do in your free time in open source you have to realize that in a business it serves a purpose like if so if if if somebody can get away without hiring software engineers they would do it like it's you everybody got to realize you're means to an end like you you know >> and it just I don't know that's a that's my heart like I'm I'm I'm disappointed by a lot of people that I previously looked up to that turned out to just be super conservative and skeptical and and not even think anything outside of the box of what programming is and >> have you seen anyone snap out of that? I mean, famously, you know, DHH, right? I interviewed him, I guess it was like a year ago, I I think we recorded it in, sorry, December last year, I think. December or November, and it came out in January.

49:21 >> Yeah. December 2025. >> Yeah. And I I interviewed it with Ryan and I nearly couldn't get a word in, but like he was like, " yeah, these agents, they're not good. It's slob." You know, stuff like this. At the same time, Karpathy was also saying this, you know, like, "Oh, it's slob. agents are slob and I was sitting there like what what what are you guys talking about like don't you see the trajectory like even if at this this point it was slob you have to see that at two months ago it was even sloppier so don't you make can you you project out like the next two months and and >> I mean they could say well at the time >> I mean look at how much worse they were back then compared to what they are now so >> you know how much ground did they actually >> lose Maybe none. Who knows? Like that's the other argument, right? That I mean now saying stuff that we've been saying for a year. Like I don't know. I think maybe we get more out of these agents than maybe somebody who's just starting now because we've seen more. We've built up more intuition, more trust and whatnot.

50:21 But >> yeah, and actually there's a bank in Australia that it's not one that, you know, we work with closely, but in my opinion, they got good and current and modern with AI too soon. they kind of hit a local maximum and then they >> started to think that they knew everything. You know, I think it's easy to think feel like you've won at that point. So I think it's so important to always have that humility to not pat yourself on the back for achieving an 85% roll out of you know x tool of >> you know >> I don't want to say what what you know kind of product you know something like GitHub copilot because the next day there could be something better that comes out and so it's about taking on a stance of constant humility and openness to change and >> yeah it's not about being right at a at a point in time. And I think one of the things that makes me skeptical of the people that have these silver bullet skill stacks is that yeah, maybe that maybe that works well today. I I I still have the question about if a skill stack is so good, why isn't it built in to the agent? I mean, that's, you know, our view.

51:35 >> Yeah. >> But it's not going to be needed tomorrow. And so, you just got to be willing to change constantly and hold on to basically nothing and move faster than ever. >> Yeah. What's your missing heart take from my list that I published? Was there anything where you're like, why I would have put this in? >> I think it's that people are not realizing just how much will change. >> Like planning is dead or it's going to change 99.99%.

52:00 Making slides, documents, a human emailing another human, most of the reasons why a human emails another human do not matter. Even if they do matter, they're going to be overwhelmed because you will not know if it's a human on the other end. I'm not, you know, I think like human to human interpersonal connection is still really important. >> Yeah. >> Who cares if it's efficient or not? Like I got a family, I got friends. Like I'm the humanity is absolutely there, but everything about how a company works will change >> and it's thinking about that maze you're looking down on. It's this this thing that's growing and it's going to, you know, go over the walls. Even if there's a wall, it's going to burrow beneath the surface. It's going to it's going to just change everything.

52:42 >> Yeah. >> And I I do wonder sometimes why it hasn't changed more yet. And then it's the stuff that we do with real customers, you know, really big enterprises. I I think a lot of people listening to this might think, oh, these guys are a bunch of cowboys. They push to Maine. They go on X. They got banger tweets. >> Yeah. >> You at least. And you know, no, we have that, but we we we temper that or or we we keep ourselves in the real world by working with a lot of big banks and and big companies and actually following this stuff through. And so, you know, we see some of why it's slowing down, but it feels like things have gotten good enough and it's moving from like there's big companies where, you know, a large portion of the people that matter the most are using agents well to now people are rethinking entire business processes with this.

53:40 >> And I don't know, it feels like there's gonna be a lot more change over the next six months. And certainly, we want to drive a lot of that. I think we got some pretty cool stuff going on. Yeah, let's see how it plays out. I mean, what we've seen is that maybe the real economy is so diverse that it kind of softens the blow, you know, as opposed to, you know, say AI was invented and it's only let loose at Google. Yeah, whatever. Like one company, I think the effects would be more dramatic than you spread it out over the whole economy where everybody has different processes. But yeah, I don't think there's a way around it. But yeah, one quick hot take to end this. I didn't put this one in. I don't think languages and frameworks matter that much more in the future. I don't know what's going to replace them, but I think the idea that I'm a Rails developer or something like like DH said yesterday in the keynote and I talked with somebody the week before where they asked me this on a podcast. They were like, "Do you think it's a matters?" I'm like, "No." Like the biggest lever that I've ever seen in my whole career for developers are these agents now. It's not a new framework. Like not even close, you know? And I do I've loved using Rails, but I think that's just over. Do you disagree or or agree?

54:50 >> I think that having a good foundation to build on is always going to be better than building entirely from scratch because it is basically just adopting the good hopefully good decisions of other people. >> Yeah. And to the extent that a framework does that and adds some guard rails or bumpers for what the agent can do or thinks it should do, I think that that's going to be beneficial. I think that we benefit from using smeltkit in our web app versus if it was just a raw node server and we reinvented speltkit from scratch. But also, we're bumping up against the edges. And I think it's now like in the last two weeks, AMP has patched spelt kit just using like PNPM patch like two or three times for bugs that it found that seem legitimate that are not present anywhere in the issue tracker or upstream. So, you know, I I still think though that we benefit from it.

55:44 >> As for Rails, I look, you know, you and I were both big Rails guys even, you know, way back when. I grew up in Chicago near, you know, where they were making rails and all that like 2005, 2006. I think it it has a great design. I want to watch DHH's video. I don't know. >> Yeah. Yeah. Yeah. I >> We'll see. I I think what matters is the the runtime behavior, right? Like we talked about this that >> Yeah. slow as >> Slow. Yeah. Is it slow or is it easy to debug or is it easy to whatever? Like that's going to matter in the future.

56:16 But yeah. All right then. Let's wrap it up. Hot take Thursday. And I think we even have we haven't even touched all of them.