Everyone is building in Claude Code. No one is running evals.

0:00 Any product person that has used observability and is looking at their traces and looking at their evals, you're probably already in the top 1% of PMs.

0:08 What is the role then of the PM? Like, do PMs need to become engineers at this point?

0:13 At the AI native teams, I am seeing that the gap between a PM and an engineer is indistinguishable.

0:19 Aparna Dinakran is the CPO and co founder of Arise AI. $131 million raised and most of the smartest AI teams I know building their evals on top of it. I feel like good eval is like you're getting some healthy percentage right, but also healthy wrong so that you can make progress right?

0:37 100%. Like I get excited when I see that evals are wrong because then it gives me a chance to know that there's improvement that could be made.

0:45 What are the things if somebody has just two hours this weekend that they should concretely go do and take away besides just they've watched this episode but now they're going to actually make impact in their career.

0:55 If you have any two hours this weekend, I would say literally what we just did right now, which is foreign.

1:04 Before we get into today's episode, I wanted to share that you can get a free year of my favorite AI tools, including Bolt, New Mobin, Arise Relay App, Dovetail, Linear Magic Patterns, Reforge, Build, Descript, and Speechify. If you join my bundle@bundle akashg.com on top of that, I wanted to quickly ask you to please double check that you are subscribed on YouTube, Apple and Spotify podcasts. It's a free thing you can do that really helps support the show.

1:33 And now into today's episode. So I've been doing a ton of episodes on Claude Code, a ton of episodes on AI agents, and separately episodes on Evals. What this episode we're doing today is we're bringing it all together for you in one iterative loop. It's kind of like the product development cycle for AI products in a single shot. So you're gonna get to see front to back how we do it.

2:05 I think we have a tremendous opportunity to learn from Aparna, so I'm gonna try to ask her the tough questions for you guys where maybe what she's doing, she's skipping some steps so that you guys can see it step by step and she's volunteered to be our guinea pig on this. So Aparna, thank you so so much for showing us the Ropes of how to do cloud code evals.

2:26 I'm super, super excited to be here. Thanks so much for having me, Aakash.

2:30 So what are people getting wrong when you look at them building cloud code agents and trying to do evals?

2:37 Yeah, I mean, I think the first question I get asked a lot is when should I even start doing evals? Like, why is that important? Do I need to think about it before I even build my agent? And I mean, if I'm honest with you, most teams are starting, they're starting with just building. You gotta start by having a real product before you wanna you run evals on it. And so today, what I'm gonna actually walk you through is the full end to end loop of getting started with building a product.

3:10 When does it make sense to actually, because of the data that you've collected, start to actually run evals and automate that.

3:17 Awesome. Let's see it in action. Where should we start?

3:20 So it's a little bit of a vision for anyone who's an AI PM today, code is so cheap to go create, which means that product taste is really the alpha today. People, especially product managers, there's all this hype around, you know, is it going to be the death of PMs? I'll tell you this, we're hiring more PMs than ever. We're hiring more engineers than ever. The ones that stand out are those that actually have an opinion and a taste around what to go build.

3:51 And so today, a little cheeky, but can we try to create taste? Can we try to have the PMs that are watching this have an upper hand to actually create that product taste? Well, where does product taste actually come from? You look at kind of some of the best products out there and what they're doing is taking in a ton of feedback. I mean, the best PMs do this. Best PMs. I mean, YC says this to every single cohort, which is talk to users and go build.

4:23 And I think what we see is that in order to actually create taste, you need to be getting feedback from a ton of different sources. It could be everything from where your team stores those issues. It could be from GitHub discussions or in real life discussions from Slack and Discord from your actual community talking to you. But also we see teams building out, really a context graph with all of this feedback. Everything from Gong transcripts every time you talk to your customers, your product analytic tools from Posthog and Amplitude and Pendo and Fullstory, even down to Twitter.

5:02 If you have A product that your users are tweeting about and sharing feedback on. These are all ways for you to actually create and cultivate that feedback source. And instead of having just a human consume it, you can actually have your agent consume that feedback. What we're going to do today is we're going to build a bit of a product feedback taste agent. This agent, you're a pm. Your job is to come in and kind of figure out what to go build.

5:32 What are users asking for every day? This product taste agent is going to tell you what your biggest pains are, what your biggest priority should be, and suggest where your product roadmap needs to go. The product I'm going to work off of today and you can pick your own product that makes sense for you, but the product I'm going to pick is actually our own open source product. Arise Phoenix. Arise Phoenix is the leading open source observability and evals platform.

6:00 You can actually get started and host everything entirely open source with Phoenix, but with Phoenix. And you're going to see what I do here is that we have a ton of backlog of issues. We also have a really vibrant GitHub discussions. We have our own Slack community, we have feedback from people who are tweeting at us. And so what I'm going to try to do is actually aggregate a lot of that. I'm going to actually try to aggregate that feedback and use that to surface up where should we go and where should we build next?

6:33 So the steps we're going to do here is actually first create this PM agent. We're going to do this using Claude code. The magic behind everything that we're going to use to improve is really tracing. We're going to, we're going to trace everything. We're going to get. Literally every step of what our agent does is going to be visible to us. And then we're actually going to run the evals. Aakash and I think this is kind of the big, you know, when people ask, when do I do evals?

6:58 You know, I always, you know, kind of point towards get the data, trace everything, get the observability. The evals can kind of help you then take you to the next level for your agent. So we're going to trace it, we're going to eval it and then we're going to do this loop where we improve our agent and bring it right back. So pick your favorite product that you want to actually use. Pick a product that you have all the context of.

7:23 You could start super simple. What I'm going to start with today is literally just the GitHub issues, the GitHub discussions and use that to actually inform what my product taste or PM agent is going to look like. Let's do this. We're going to go ahead and build a PM product taste agent just using cloud code. So go ahead, kick up cloud code in your terminal for product folks. You know this might feel intimidating in the beginning, but I can guarantee you the level of control and iteration you're going to get by just doing this in your terminal and getting comfortable is going to feel just the unlock you're going to get is going to be worth a little bit of that learning kind of pain in the beginning.

8:06 I'll be honest, I've not always been the best with my email inbox and just thinking about it made me feel anxiety. But my anxiety has really never been lower since I started using Superhuman Mail Today's podcast sponsor. Their Ask AI feature is one thing that really stands out for me because I have so many contract details or deliverables buried 8 replies deep and I can just ask the AI. I also love the auto drafts feature so that I have a draft to react and respond to and of course their follow ups and are a lifesaver.

8:33 Now is the time to give it a try. Check it out@superhuman.com Akash Today's episode is brought to you by Vanta. As a founder, you're moving fast toward product market, fit your next round or your first big enterprise deal. But with AI accelerating how quickly startups build and ship, security expectations are higher earlier than ever. Getting security and compliance right can unlock growth or stall it if you wait too long. With deep integrations and automated workflows built for fast moving teams, Vanta gets you audit ready fast and keeps you secure with continuous monitoring as your models infra and customers evolve.

9:08 Fast growing startups like LangChain, Rytr and Cursor trusted Vanta to build a scalable foundation from the start. So go to vanta.comakash that's V-A-N-T-A.com A-K-A-H to save $1,000 and join over 10,000ambitious companies already scaling with Vanta.

9:29 So let's do this. Go ahead and create a repo or create just a directory and you can go ahead and initialize Claude inside of that directory. Let's just go ahead and first give it a starter prompt. To actually build this agent I'm going to ask it to build me a PM agent for the arise AI Phoenix product and I can go ahead and actually just link the URL to that entire repo directly in here so that it has exactly context of what I'm asking it to build.

10:08 Then I'm just going to go ahead and ask what context do I want it to happen. Pull recent GitHub discussions, pull all the recent releases and look at the GitHub issues. I'm going to start piecemeal here first versus starting with contacts from one location, which is GitHub. As we scale this, you can add in contacts from like I was saying, your gong transcripts, your product analytics. You can add context from literally your Slack convos, your discord channels.

10:43 Anything can be brought into here. And what I first wanted to do is first just figure out score the issues and the discussions based off of priority. First just figure out how important is the stuff that we want it to actually look at and build. So things to look at is like bugs versus features, reactions that people gave it comments. I do want it to look at recency. So these are all things that I'm actually asking this product agent to take a look at and consider.

11:17 Then call Claude or you know I can be specific here. I can say call Claude opus whatever model I want. So call Claude with. You know, I can even ask it to go ahead and do some kind of like prompt caching so that it doesn't keep pulling down the issues every time that I run this loop. But just to keep it simple in the beginning, what I'm going to do is just call Claude and write down a just a markdown PM report that has, you know, that has as the output the top pain points feature asks and themes.

12:08 Order this by P0 to P3 priority. This is basically going to be like initial starter prompt for me to actually build this product taste I can get super Typically what I like to do is be really thoughtful about the plan that I'm giving my agent so that it it's not just going off of nothing, but there's also times where you'll just have it go off build something and then you're iteratively giving it feedback and that's totally also okay.

12:39 And then I'll just say here, use my GitHub token and my anthropic API key. So let's see what it can come back just with that super simple while this is going and kind of doing its thing in the background, but I'm actually going to show you is you can see it's going to interrupt and ask a ton of questions as we go through this. But what I'm actually going to show you all is just a simple one I built right before this and see if we can get the one we're building right now to just match up and see how, how close we can get in just an hour here.

13:12 Okay, so this is basically a PM agent that is already built out and already kind of, you know, we've had tracing set up and ascending to arise already. And I'm just going to open one of these so I can show you all kind of what it looks like here. But this PM agent is. These are the traces of our actual PM agent. And for those of you who are like, what's a trace? Like, that's, that's, you know, new concept to understand.

13:44 You can think about a trace really just as it is the step by step playback of what this agent actually did. In this scenario, this agent is first going ahead and pulling back GitHub discussions. It's pulling back the GitHub issues, it's figuring out what are all the releases that were recently released. Then it's going through and it's actually looking at every single issue that is inside of that project and it's actually consuming all of these and coming up with a score of how important each of these issues that it's raised are.

14:27 As a product person, this is kind of the first thing you need to understand is like, how important are all of these asks that are coming from your users? What is the pain that it's solving? And so the first thing I'm just asking you to do do is figure out, well, can you score basically how important is each one of these asks that are coming back from for this project and what I'll actually do, you know, as, as it scores, I want to actually have an eval that will evaluate how good was the score that my PM agent actually came up with and is it accurate or inaccurate based off of the context that I have around how I want to prioritize bugs, how I've historically prioritized feature requests.

15:15 And so I actually want to write an eval that will help teams kind of evaluate the quality of this initial PM agent that we've built. Go back and check on our agent here and see how far we've gotten. So, still thinking when somebody's setting up

15:35 this repo correctly, basically you created a new GitHub repo, you gave it your anthropic API key. I guess to create the repo, you have to log into GitHub. Those are the main steps people have to do before this.

15:50 Correct, Correct. I'm happy to go ahead and send you guys no or sample repo if you want to get started doing this yourself so that you can follow along with a project of your choice. But in this case you can see. Great. Okay, so it's gone ahead. It's actually built this agent. I'm going to go ahead and it just looks like it's updating. What the. What the. Okay, great.

16:21 So it's actually just updated. It's using my GitHub token, it's using my anthropic API key, and now it's actually going to go ahead. It's pulled 40 discussions, 60 issues, eight releases, and now it's going to go ahead score each item and then based off of the score that it gives every single one of these issues, it's going to go ahead and give me a report about what the most important things to actually top pain points, feature requests, themes, what shipped, and give me a game plan that I can then start use as a starting point when I come in a really useful feature that you'll do this once today, but ideally you want this kind of running all the time consistently.

17:04 Every time someone adds a new bug report, adds a new issue, it's always doing this. What you can do is actually just say, can you run this in a loop? Can you run this in a loop? And you can specifically say using the Claude loop kind of skill. This is really awesome because what Claude does is that it spins up. Essentially a cron job. Well, what's a cron job? It's basically you asking Claude to be able to run some type of workflow that you do every day in a loop and in my case, every day, every hour.

17:47 You could set this to every five minutes if you wanted to. It'll go ahead and it'll go ahead and actually run this loop every, you know, however cadence you set so that it actually does your job. Every hour you have the latest report of what you should be prioritizing for your agent. So let's go ahead. Oh, it looks like I need to go ahead and set my GitHub token. So give me one second and let me do that and then we can actually go ahead and run this agent and you can watch it live.

18:21 So this is actually going ahead and running my Phoenix PM agent. I'm going to show you guys how to do this so that you can also do it. But I've also kind of already set up traces. What does that actually mean? Tracing is the way for teams to actually get visibility into everything these agents are doing. This is a really hard thing to debug because Claude is spinning off a bunch of different things and running this in a loop.

18:59 And you might not always know if it comes back with slop or it comes back with something great. How do I go and improve it? How do I go and figure out how it did that? Tracing is a really awesome way to understand what your agent's doing today. What I'm going to actually show you is that I'd say tracing used to be really hard. You had to go call your engineering partner to have to go and set up tracing.

19:24 I think with AI, it's probably never gotten easier to do this. So what we have is essentially skills. We've released a kind of a series of call it skills that you can actually just give to your coding agent. This is kind of a set of ARISE skills. You just go in, install NPX skills, add. I'll show you. We'll go ahead and do this.

19:54 But. But once you actually add this, you can just ask Claude Code to go ahead and instrument the entire agent that we asked it to go build. Right now you're looking here at a whole bunch of different skills. One of them is the ARISE instrumentation skill. For those of you who are curious, it's literally just in English, telling what CLAUDE code should do to actually send trace data over to Arise.

20:25 It makes it super easy. I'm going to show you. It's going to feel super magical. And you're not going to need to wait for your, you know, you're not going to need to wait for your engineering partner to have to go and do all of this lift to go get data from your agent to your observability platform. So let's go do this. What we're going to do actually is from here I'm going to say, can you help me instrument this agent?

20:54 So I'm going to go ahead and actually ask it to instrument this agent. So what this is actually going to do is call the ARISE kind of instrumentation agent. So you can see here, sorry, the instrumentation skill that we just talked about. So it's going ahead, it's calling the skill. This instrumentation skill will actually first look at the code base and understand how is this agent built, what's actually calling the LLM calls, what's actually calling the tool calls.

21:35 And it'll go ahead and it'll figure out kind of, you know, this case, the language that it was written in as Python. The LLM provider was anthropic. Here's the library to go use. Here's what it's actually going to go do to set up the different calls and it says, cool, everything is already wired up, sending to Arise and is there anything else specific you'd like to go change? So now let's go ahead and just see run my agent, see if it sends recent traces and I should be able to go pop over to the platform, my observability platform, and go look at traces.

22:15 We'll see if there's ones that are going to show up right now from my recent run. But it should go ahead and actually start streaming in traces from the last area. This is everything from the last 15 minutes that's just showing up here. You basically get a way to do all of this and it figures out everything from here's the individual alum calls, here's the actual tool calls that were made, here's the.

22:47 It had to go and fetch stuff from GitHub, it had to go score every single individual LM call and then it finally had to come back with that report that I asked for, which was what are my top pain points? What are my top feature requests? What was already shipped? And so you can see here, it's giving me an executive summary, my top pain points and kind of the things that it scored really, really highly for me to go and prioritize for my product.

23:19 Literally I didn't open any ide, I didn't open anything. I literally just asked Claude code to build me an agent, gave it a really good prompt, and then I asked it kind of what I was hoping for and then I asked it go instrument my agent with arise using the skill and boom, now I have visibility into my agent. Everything's probably not going to be perfect and I can probably already guarantee you that, that it's not going to be perfect.

23:54 But what we can do is actually start using this as a way to understand, well, how would I improve this agent. What I'm going to actually show you right now is actually an in product agent that we've built called Alex. Alex is an agent that sits inside of our product and you can ask all sorts of questions like help me figure out the common types of issues that are coming up.

24:29 This will actually go through. It'll look across all of the data, like the inputs and the outputs, and it'll start to surface up common types of issues that users are asking from my traces. I can use this to actually first figure out what types of evals should I actually be running on top of my agent. And the reason why that's interesting is that you're starting your evals from a place of actually looking at your traces, looking at your errors, and trying to understand, well, did it actually score some things correctly?

25:04 Did it not score some things the way that I would have prioritized things? How many times have you had someone on your team kind of say something was super important, super priority, but you wouldn't have given it that high of a ranking for yourself? The next thing that I really want to show is really for teams, is how can you use Claude code to actually help you figure out a baseline eval for these agents that you're building?

25:29 You can have it start, just build a baseline eval and use that to actually iteratively improve your eval so that you're not starting from complete. So scratch. So what we can do here, you can do this in our product. You can also do this, you know, kind of, you can also do this using Claude code. Again, so kind of in the theme of today, I'll actually do this using Claude code and show you how you can set up evals directly from your terminal.

25:59 But you're going to see here, once I have the traces sent to arise, I can actually ask, can you suggest a good eval for my agent? I want it to. I can just start with that. Can you suggest a good eval for my agent? Let's see what it comes back with. What this will actually do, it'll call the skill the evaluator skill that actually looks at, looks across the traces and suggests kind of there we go.

26:38 Okay, so looks across the skill and it suggests, okay, well, these are kind of three evals that you might want to do. There's report groundedness checks, whether the quotes, the issues in the final PM report are grounded in the actual data fed in. It runs kind of across everything. So it's almost, you know, I think about this almost like an eval on the final report that was created. You could do an eval on priority alignment checks, whether the p0 p1 kind of in the report matches the top scored issues from kind of what you're expecting or something around report actionability.

27:21 Okay, well, I could do these, but these are all things that are kind of looking across almost like the end product. What I actually want is something different as a pm. What I want is actually to look at every single. I want to get a little bit more granular in the beginning and start to understand for every single issue that this kind of, for every single one of these issues here did it actually give it a right score.

27:49 Like in this case it said that, you know, it gave it a priority of a 3. In this case, I don't know, let's, let's pick another set of them. This case it gave it a zero. It said this integration is not that important. It gave this Privacy Question A3. And so there's kind of all of these. It's kind of making up these priorities. And I actually wanted to first just evaluate is the score that it's attaching to kind of determine how important these issues are.

28:20 Is that actually something that I would have set by myself? So I actually wanted to run something like a priority score. A priority kind of eval on is the score that it's actually saying how important these GitHub issues are. Are they actually accurate based off of how I want to weight them? So let's go back to cloud code. I can actually just ask it to help me come up with a way to eval this.

28:51 And this is very normal where you're kind of doing this back and forth with Claude and you're actually asking it to go back and repeat yourself and get really specific about what you want. So in this case I can ask, can you help me build an evaluation to evaluate if each issue is actually scored correctly or it's each issue's priority.

29:23 Maybe is a good way to say this. Each issue's priority is actually scored correctly.

29:30 I think that's option two, right? Priority alignment.

29:34 Yeah, yeah, this is. Oh, well, this is, this is slightly more about at the end because it looks like it's checking at the end in the very report if the top scored issues are kind of what I would have picked. But what I'm looking for is something slightly more nuanced, which is not just the top issues, but every single kind of individual issue is actually given its appropriate weight.

30:06 It's giving me this priority accuracy evaluator. It'll go ahead, it'll create a way to run this evaluator on top of the actual traces. In this case, it's already picking one that I've actually already created to do this just to show you guys kind of how this works, but it'll kind of suggest, hey, there's this eval you've already created, which is kind of doing this like row level, issue, level kind of priority.

30:38 And then it's actually going to use this to go and run it on top of those kind of traces. So in this case it's saying, hey, it's running it on older data. Do you want to Go ahead and run it from today's issues, like the new issues that you just grabbed from today. So it'll go ahead and start running it on the newer spans. You can see here every single kind of GitHub issue that has come in, it's going to go ahead and give it a score of how important it actually is.

31:09 Then it'll evaluate whether that score that it was given was actually an appropriate eval or not.

31:15 I hope you're enjoying today's episode. Are you interested in becoming an AI Product Manager? Making hundreds of thousands of dollars more joining OpenAI an anthrop then you might want to do a course that I've taken myself. The AIPM Certificate ran by OpenAI product leader Mikdad Jaffer. If you use my code and my link, you get a special discount on this course. It is a course that I highly recommend. We have done a lot of collaborations together on things like AI product strategy, so check out our newsletter articles if you want to see the quality of the type of thinking you'll get.

31:47 One of my frequent collaborators, Pavel Hearn, is the Build Labs leader. So you're going to live build an AI product with Pavel's feedback if you take this AIPM certificate. So be sure to check that out. Be sure to use my code and my link in order to get a special discount. Here's the dirty secret about prototyping. You spend two weeks building a prototype. You validate your assumptions. Engineering loves the direction. Then what happens? You throw the whole thing away.

32:10 Bolt changes this completely. When you prototype in Bolt, you're not building throwaway mockup. You're building real front end code that integrates with your existing design system. So when you hand it to engineering, they don't throw it away. They they ship on top of what you've built. I use Bolt every single day. I host my land PM job cohort on it. And honestly I'm up till 2am some days just vibing in the tool, having fun and building.

32:33 That's when you know a product is good. When you're using it past midnight. Not because you need to, but because you want to. Check out Bold at Bolt New link in the show Notes I used to think I had a retention problem. Turns out I had a messaging problem. I was sending the same onboarding emails to every new use, whether they activated on day one or never logged in again. I had no idea who was slipping or why customer IO changed that.

32:54 Every message I send is now based on what users actually do in the product. Someone hits a key Activation moment, they get nudged to the next one, someone goes quiet, they get a different path entirely. Their AI agent makes it fast. I describe the campaign I want and it builds the full journey, form triggers, timing, copy, even branching, logic. And when I want to know how something is performing, I just ask the agent directly, directly, and it tells me what to do next.

33:17 They also have an MCP server, which means AI tools like Claude can see directly what's happening in your Customer IO workspace, your segments, your customer data, your attribution, all of it. So instead of explaining your business context every time you need help, Claude already knows it. Notion. Use customer IO to personalize their onboarding and hit nearly 50% open rate. Improved conversion by 6 to 7% with localized campaign campaigns and pushed open rates up another 20% through AB testing.

33:46 The idea is simple. Customer IO helps you deliver more impact from every message you send. If you're a PM or founder and your onboarding is still one size fits all, try Customer IO at Customer IO. I'm keen to see what evals it creates. I guess the traditional sort of evals teaching literature is all about, like, you finding production traces that you feel like there was an error. So I guess that thinking would say you'd go to the trace dashboard in Arise.

34:18 You look at those priorities, you'd say, oh, this is a zero, but this really should have been a four. And you'd pick up like 50 of those errors. Then you'd group them and say, like, okay, These are the 10 errors that it does. So are we trying to replicate that process but have Claude code basically do it itself? Is that what we're doing here?

34:39 Exactly. Basically what Claude code is doing is it has access to all of the traces in Arise because the skills, basically it can go and call an API and I can share what it's doing under the hood so we can talk about it. Because it does feel a bit. A little bit magical when we just talk through it. Give me one second, Let me share the secret sauce of what's happening here.

35:11 Under the hood. All of these skills are actually calling APIs. Specifically the APIs that skills tend to call. Is that what we've realized is that these coding agents are really good with command line or CLI interfaces. So what it's doing is basically under the hood calling and fetching all of the traces. You've seen him and Shreya tell you, hey, go through line by line, look at where the individual traces failed.

35:49 That is totally a great way to do this. You can, of course, go in and get started and start doing annotations and start doing. Did it actually answer the question is this. You can write freeform text and just write freeform text about what was good, what was wrong about this. It's absolutely a great way to do that. I'm also someone who I love to see if cloud code can help me cut some of that time and surface up some insights for me.

36:22 And so what I'm actually doing here is trying to understand just with Claude code and if I can give it access to my spans and my traces, like, what are some insights from this that I should have to go and learn, you know, help me go and tell me what's wrong with my agent. And sometimes it's, you know, just being super honest. Like, sometimes it might not come back with something amazing as your first eval, but what I typically like about it is that it gives me a place to actually start thinking about problems and start thinking about areas of improvement.

37:01 So in this case, I've gone ahead and created this, like, priority accuracy. Like priority accuracy eval. And it's running, it's now running. It's run across all of my new spans. And I can go in here and just say, show me everything where the label is actually inaccurate, where Claude code thinks that the priority, you know, you can see the scores here, the priority that it's come up with is actually wrong. And why is it wrong?

37:31 And, you know, this is probably something that you're going to hear all the time from folks who do evals is, was my eval wrong or was my agent wrong? And you will definitely have scenarios. And there's a whole process that Hamel and Shreya actually talk a lot about, which is aligning your evals so that your evals are grounded in that kind of human feedback. What I'm sharing is kind of a way right now of, can you start.

37:58 It's almost like, can you start with the vibe eval and then modify it and improve it so that it becomes something that you can trust and go from? And you can do either approach. You can go through the axial coding approach, surface up all the issues, have the human in the loop, and identify categories of pain. But as a product person, you might already know what types of things you definitely want to catch.

38:33 For me, what I want to catch is, is every single issue that this agent is prioritizing, is it right or is it wrong? Is it accurate? Is it giving it an accurate score or is it not giving it an accurate score? And I can start off by saying, well, Let me see if I can just have it go and create an eval to suggest kind of what that priority accuracy looks like. You can do it through a skill.

38:57 You can also, if you do have human annotations that are built through here, the skill will look at those human annotations and use it to actually build you an eval as well. I, in this scenario, didn't have any, but if I had one, it would go through and do the whole process that Hamel and Shreya kind of walked through of aligning the evals. So it's gone ahead. It's run the priority accuracy evaluation. It's comparing the accuracy of.

39:27 It's something that's looking at the score that was assigned to each of the issues, and it's surfacing up. Is this an accurate score or is this not an accurate score? Again, this is just based off of a simple first pass of this eval. I am going to refine this eval now because this eval is completely based off of just Claude looking at my traces and trying to identify problems. And the whole point of this is like, how do we get this loop kicked off?

39:58 This loop is meant to kind of give you a starting spot. It is not meant to be your end all, be all kind of state for your evals or your agent. Your evals will adopt, will kind of get better, and your agent will get better. And that's kind of what we're showing in this workflow today is kind of how do you get started, how do you get unblocked? And then how do you do that improvement loop so we can make this better?

40:23 So in this case, I kind of have a very simple small eval here, which is okay, looking at the accuracy of the score. These are ones that clog. Things are not accurate. I can actually just directly ask here, like, when my priority accuracy is inaccurate, what are common issues or reasons for that? So this will actually kick off. And now look at, well, what types of things is my PM agent not prioritizing correctly?

41:00 So I have my agent kind of kicking off looking at the data. And what we're trying to do here is really go from you built an agent, you have traces set up automatically through code. Claude. You have Claude kind of suggesting what an eval could actually look like. And now these are already scenarios that Claude thinks are not right, accurately scored. This is a great starting ground for me to say, okay, well, what can I go to understand how to go improve this agent?

41:32 And you barely had to write. You didn't have to write Anything. You kind of had to ask Claude a couple things. So let's go ahead. And this is Alex kind of giving me suggestions of kind of what to go do here. So in this case, there's whole categories of issues that it's looking at. So there's some. Where there is a feature request scoring, there's a legacy scoring system, there's bugs priority scoring, there's low priority scoring, there's data fetch.

42:07 Okay. There's a lot of different categories of where it's actually suggesting that my scoring might be off. And it's giving me a whole bunch of spans to go look at, to go debug and understand what are some actual problems that this PM or taste agent might have in prioritizing issues that are coming.

42:26 And what's a span exactly? It's a group of traces.

42:29 A span is really an individual step in a trace. So in this case, what you're looking at here is this. Is this entire interaction where it did this whole report is what you'd call a trace. A span is a single individual step or a single individual issue that I had to go look at.

42:48 Got it.

42:50 Yeah.

42:50 And it's weird, isn't it a bit weird that Claude rated like everything it did inaccurate.

42:56 I. Some of them are accurate. And some of them.

43:00 Some of them are accurate.

43:01 Okay. If it did, then that would probably be a good spot for you to understand. Okay, well, maybe I shouldn't trust that eval from. From Claude. And so I feel like a good

43:12 eval is like you're getting some healthy percentage. Right. But also healthy wrong so that you can make progress. Right.

43:17 100%. And so you want that feedback of like, I get excited when I see that evals are wrong because then it gives me a chance to know that there's improvement that could be made. But when everything's wrong, then, you know, it's obviously that's definitely a scenario where you need to start looking at your eval to understand what. What to go improve.

43:39 And when. When can we do the vibe evals? When do we have to do the axio coding? Or can you always start from vibe valves and then layer an axio coding talking to the agent later?

43:51 So my take is that vibels are going to fall short very, very quickly. It's in. The reason for it is that it just doesn't have any. It's not grounded on any actual human that is involved in curating that taste again, of your agent. And so what you really want is something that helps you. You know, I think that it Would be hard to say, hey, you have to go and immediately start by, you know, having a bunch of vibe evals and using that to evaluate your agent like that.

44:34 It just the signal to noise ratio there is going to be really, really low. And so having something where you have maybe a simple thing that gets kicked off. But then now what I'm going to actually go do here is that process where I have a simple eval and I'm now going to make sure, okay, well, is this eval that I've created actually something that I can trust? And it's not going to be. It was a one shot eval that's out of the box.

44:59 I'm going to actually go through and figure out, well, where do I disagree with it? Where do I not disagree with it? How do I actually. And you would do this process even if you did axial coding, even if you did axial coding and you did individually human annotated every single span and every single issue and you were able to put together this amazing ground truth data set. Your eval will get misaligned over time as you see more and more data.

45:28 And so it is super important that you regularly align those evals to the data that you're actually seeing on the ground with your users. What I'm going to do right now is actually walk through a process where I've created a very simple eval out of the box. Claude just one shotted it for me. Now I'm going to start asking, okay, well, is this an issue with an eval? Is this an issue with my agent? And you have examples in this scenario.

46:01 It looks like. Bug category items using the new scoring system with category 4 are also commonly inaccurate. It feels like there's scenarios where bugs maybe are not getting categorized or given the accurate score that I wanted to. In my world, I want bugs to always be super high because if it's a bug and a customer hits a bug, that's just a really bad experience with the product. I would prioritize bugs over even new feature work.

46:33 This gives me a way to say, okay, well let me go look at some examples of where the bugs are being prioritized really low and just gives me a category of problems to start looking at and start debugging and understanding how good this agent is. And what you can do is for some teams, these evals end up as they get really good and as they get really better. You can immediately ask Claude to going back to using Claude code with evals, say, hey, go grab everything where this eval failed and suggest an improvement and go improve that eval.

47:14 For me, I think it's unfair to say people aren't using Claude to create evidence. And I think that's maybe one of the pain points that I see with always saying start axial coding is that in reality you will always do it. But I think it's okay to start with Claude suggesting what a good suggestion of an eval could be. And these models have gotten so good, like having it go through and look at your answers and suggest, hey, that probably is something you should flag and look at.

47:44 I would trust it. I would trust it as a first pass. Like go tell me what my email valve should be.

47:50 Yeah, that's my favorite workflow. Always start Claude generating it. But then you just give it like ruthless criticism and I just turn on dictation mode and I'm like, well, you misjudged this for this reason. You must judge this for this reason. And that's where the taste alpha that you bring can actually come back in.

48:07 Totally. And I think what for me is like, how do I quickly get into that loop is get data in, get an eval set up, give it criticism and let it go run on a loop. So I showed earlier there's kind of the Claude loop kind of skill that Claude has. And so what you can actually do here is now that you have this eval, you can create a whole other skill that's just like every day, go through, fetch everything that was inaccurate and go that was inaccurately prioritized and go fix and improve my agent and you can go and create a skill that actually will then go suggest improvements for your agent from the evals that you just ran on top of this.

48:55 So you actually loop the improvement too, not just the agent.

48:59 Because then you get to a world of self improvement and that's where to be honest, I think we're all headed is that the data that we all collect, the evals and observability is the foundation for self improving agents. And so you get your observability in, you build an initial eval, It's a first pass. You're going to make it better. You're going to have to give it ruthless criticism to kind of make the agent better or make the eval better.

49:26 And I think what teams are doing right now is they're kind of doing that iteratively. You can just create a loop that essentially starts to look at the evals identify. I just asked right there, give me the common reasons why the priority accuracy is inaccurate. Oh, it's because the way I prioritize bugs doesn't look right. What I can do, go back to my PM agent and just say, hey, go fix this issue. Then go fix the issue.

49:56 Ship a new agent now go collect traces from the next rev of that agent. That improvement loop can actually run inside of Claude code as a loop skill.

50:08 That is all fine and dandy for your internal agents that are assisting you in your work. How does this all change for the AI agents in your product you just showed us, Alex? Maybe you can go under the covers of how that worked when it's actually a product. You're not going to be shipping self improvement to Alex every day because you don't know, it could just go off in some weird direction. So how does, where do the human in the loop parts come in there?

50:34 Totally. I mean there is, there's still code review, there's still a human that, you know, looks at every PR that is actually being put up by this self improvement loop. But you know, maybe what I can ask you back is, but isn't that the vision, isn't that the future that we all want to go to is that I should be able to see someone file a bug and on, you know, Alex didn't give me.

51:06 Somebody gave a response that Alex gave a thumbs down. Alex is able to immediately. And this is kind of what we're doing internally already is you're going to hear a lot more about us talking about, about it in the next couple of weeks. But Alex has already taken that feedback, spinning up a whole debug kind of workflow and using the eval, using the trace to debug what went wrong. And then in some scenarios, like we talked about, it's the eval that's wrong.

51:37 And in that case it's a refinement on the evaluation. And in some cases that's great, right? That's basically a little bit of what you hear all about the axial coding of figure out what are the reasons why that eval wasn't good and then use it to go improve that eval. In some cases the eval was right and it really was the agent that needed to be handle a specific scenario better. In that case, what we can do is just very simply go in and do an improvement, say, hey, go fix this and actually go in and improve the agent.

52:15 What I can do is we want, ideally it's happening in real time across millions of users automatically, I guess how do you do that safely? Code review is one step. What else do you need to. Where do you need to put the human in the loop.

52:29 I think there's a couple maybe places where that needs to happen. One is as the eval changes, that's also a really important step to actually having the human kind of curate that taste of what is good and what is not good. So the human is typically involved in eval changes, they're involved in the agent changes. There's a lot that's happening right now around making sure that the skill that's actually being used to do the improvement workflows, that is one that is typically designed by a human.

53:11 What does that improvement skill need to look like? What is all of the context that it needs to have access to in order to be able to know what the improvement is? In this scenario, it might not have all the context because all I gave it was just GitHub issues. But if I could then layer in my product analytic metrics, I could layer in my traces, my actual entire traces, it could actually end up using that information to build its own context of what went wrong.

53:46 How do I need to go fix it and leverage that information as well? Basically context for the improvement loop.

53:53 Got it. So there's human in the loop at any agent change, any eval change. But outside of that you can actually use loop commands within cloud code or whatever if you're in more production database, a real cron job and every day or whatever cadence. And so what you get to work with like all of the best companies, uber doordash, you name it. What are the, what is the state of the art looking like for this self improvement?

54:18 How fast are people moving and how fast do they need to be moving to be competitive?

54:23 I mean, I think it's going to come very, very quickly, if I'm honest with you. I think the best teams are already doing this in their, you know, call it like a radius that they're comfortable with today. But that radius is going to get bigger and bigger. Is there, you know, maybe the initial improvement is around improvements to the agent that are kind of more simpler, more around the prompts, the tools. Does that radius then become about giving entire workflows that the agent didn't have access to do?

54:57 Does it? So the radius of those changes I think is going to become, is going to become increasingly bigger, which we're excited about. But it's just that self improvement loop is not going to happen without having really good data, really good data and really good evals. If you think about. And just to try to maybe take an analogy for something that's so different. But if you think about like some of the best sports players, what do they do?

55:30 Like I'm talking about like the Nidols, the betterers. If you're a tennis fan, like your Novaks, what they're doing is actually looking at their plays. They're looking at their previous games. They're looking and studying their behavior of what they did and using that as a way to understand what went well and what didn't go well in their games to go make improvements. This is kind of studying your plays is kind of what agents, you know, self improving agents or self improving harnesses have to do is they kind of have to study their own plays to understand what did the human say was a good response or what did the human not say was a good response and use that to actually figure out how to improve their own gameplay in some way.

56:20 And that's what we're actually, that's why the evals and the observability are kind of the foundational layer in order for teams to actually build that self improving loop.

56:33 So I personally have encountered PMs that I feel like are in one of three buckets and I think you have customers in all three of those buckets. So there's the AI native like customers you have like Handshake and the AI companies. Then there's like the digital first companies customers you have like Uber and Reddit and Roblox. And then there's like the normal companies who have tech arms, Pepsi, Conde Nast, normal type of companies. So you get to, you work with all three of those groups.

57:00 And so what I want to understand is usually the AI native groups, they're going to be doing the quote unquote best way or like the right way of how to do things. So what are the AI native groups doing? And specifically not just with like how they're building their evals, but the role of the pm. What is the role of a PM in an AI native company versus a company who hasn't gotten there yet and how does that company bring their PMs there?

57:26 Yeah, I, I think the role of a PM is like completely changed in the last year. The role of the PM is almost like the, you're the taste maker for this product and in order to become a really good tastemaker, you really have to understand the outcomes of the agents that the, especially the AIPMs, where the product is the agent, the product is the agent that's being built. You have to spend a lot more time. The AI native PMs, they are almost indistinguishable from engineers in some ways because they're comfortable living in cloud code.

58:07 Like this entire workflow that I just showed where they're able to build even just a simple internal agent to help them do their daily tasks where they can. And you're not doing things. We kind of say this internally and I think it's true. It's like if you're doing things the same way you were doing things last year, then you haven't caught up yet. And I think that, I deeply do think that if you're kind of looking at your old board of like here's my priorities and you're kind of manually scanning them and manually kind of understanding every single kind of doing what you used to do.

58:47 It's just different because now with the advent of cloud code, I can actually have it. You're not limited by how many individual meetings and gong calls that you can personally kind of hear. You can have cloud code go through and it has access to all of these customer calls that you might have never been able to consume, consume all by yourself. But can it help surface up the one or two that are like super critical? You need to put your eyes on that because that's going to help you unlock your next 10, 15 customers.

59:20 And so I think in these AI native companies what we're seeing is that the PMs were able to leverage cloud code to do everything from understand user data and users user feedback better surfacing that back into what does a really good product experience look like, get really close from idea to solution. So it's not like, hey, I'm handing it over to an engineer. It's like they're able to effectively almost put together a plan for what that build needs to look like.

59:57 Those are, Those are the PMs that I think are really going to be 10x or whatever multiplier PMs in any, in any team.

60:06 So we're talking about working with those AI native companies. You are yourself one of the AI native companies and you refer to this that you yourself are hiring more AI PMS than ever. So what does the new profile look like if I want to land an AIPM role at an AI native company that has raised $131 million? What are the skills I should be developing? What is the depth of technical knowledge and topics I need to cover?

60:35 1 and I have always believed this is like just the curiosity is the number that's like for me the number one most important signal. Like this person is trying all the new tools. They're kind of Exploring the boundaries of what they can and can't do. Because that's something that there's kind of the old way of doing things is that there used to be trainings and you'd go to these trainings and someone would walk you through how to use a tool.

61:08 What if the tool is Claude code and it's had shipped 90 features in 30 days? There is no old way of doing things where you can have like a daily training for a product that's moving that fast. And so it's kind of the onus of keeping up has become on the individual now to actually keep up with the tools, keep up with what's changing. And if something, not everything is going to be useful to what you do, but if something can give you an ability to, hey, that used to take me an hour, now it can take me 10 minutes.

61:44 Like that is an advantage. And being able to identify those and use them to your advantage is deeply, deeply. I think it's built off of curiosity at this stage. I think too the other big one is it's still really important to care and understand the user. And customer empathy is something that I don't think AE like the best PMs and the best product tastemakers, you know, understand.

62:18 You could ask them, how is that customer using the product, what's their biggest pain points, what do they, you know? And they would be able to rattle them off to you. And I think what's now changed is that you can actually get even deeper. You can, you know, customer asks for something, it could have taken a week to go build that, two weeks to build that in the past that could be delivered that day if you're able to ship at that velocity.

62:46 And so being able to get even closer and deliver to customers even faster is no longer just like a. It's no longer a pipe dream. It's actually how the best products at AI natives are shipping right now.

63:04 So 99% of people aren't in an AI native company. So they don't believe us. So I need to just confirm this is true. What you're saying is that sometimes an issue will come in, your PMs will identify, it's important enough. Either they will prototype or an engineer will prototype and make ready for production a feature and you guys will ship it in the same day. Yes, that is actually what's happening, guys. She said it herself. What is the role then of the PM?

63:35 Do PMs need to become engineers at this point?

63:39 I think that at the AI native teams, I am seeing that the gap between a PM and an engineer is indistinct distinguishable because when code is become so much easier to actually produce, then actually, you know, this goes back to where we started today's podcast with which is the alpha is the alpha today is product taste.

64:10 So the people that understand product taste, understand what customers want, understand how to deliver a really amazing experience, are just going to have an insane, insane velocity. So PMs who can kind of go from here's the pain point, here's what I would, I think is a really amazing experience and they are a triple threat where they're like, I could probably go build that today and figure out what that, you know, talk to cloud code and figure out what to go build like that is, you know, it's, it's a triple threat.

64:46 In this, in this environment right now,

64:49 what are you seeing at the enterprise level? Because they're not even close to there. So if you're at a big enterprise, if you're at a Pepsi or something like that, you're still trying to take on the best practices. What realistically, what can they take on and how do they take them on?

65:05 Yeah, I mean I think what I'm seeing in enterprises is like they're still innovating at there's, I don't want to say that there's no innovation happening there at all. Like right now all these teams are all using the coding agents and I think feeling the unlock of those tools in their own day to day workflows.

65:37 I, I think what I'm seeing coming out of the teams right now even there is like one, amazing products that use AI to make the experience of that product useful. Two, I think there's usually a massive, especially larger companies, you have silos of data and people who might have access to some information and other teams don't have access. And there's actually a really great piece that Jaya Gupta, someone you should follow on Twitter, kind of shared a couple weeks ago.

66:13 Now that's gone super viral around context graphs. And what a context graph is is essentially can you give your agent access to. The agents are only as good as how much context that they actually have. And then of course the harness that's built on top of that access to that context. And so instead of all that information and data being in completely different silos and people operating in these silos, can you give one unlock for agents is that, can you give it access to context from different environments?

66:49 And what that does is it actually makes people kind of bridge the gaps across, across different teams in ways that probably weren't possible before. And so figuring out how agents consume the context within an organization is going to be probably one of the biggest problems. I mean it's probably one of the biggest unlocks challenges and unlocks that we're going to see this year.

67:16 So if you're a product leader at one of the enterprise companies, you're seeing what you just demoed for us, you're saying, okay, how can I bring my company towards that? What's sort of the step by step roadmap I should be implementing over the next say 12, 24 months?

67:32 Well first I think as an individual I see, I think build like building and what I just shared right now of like you'll read a lot of just stuff on AI Twitter of everyone kind of, you know, everyone kind of sharing every latest new model and every latest new tool out there. I think what I would just highly recommend for any IPM is start by building. Start by building very simple. Like this example that we just did today.

68:03 It doesn't even need to be an external facing agent that you need to publish. Can it just be an internal kind of tool that you use use to actually help you unlock, make one big unlock today? That's huge. That's huge. Because think about, you know, if this tool that we just vibe coded in an hour now I'm going to go use it to figure out okay, well what are my top pains? Like what are this like you can imagine the next step after that is well, can I get an agent to actually go and put up a draft PR for one of these?

68:38 Can I get an agent to actually then review that PR and do the code review on that and then you know the process to go from identifying a pain point to solve and then releasing that could have taken months in the past can now that entire thing can be shortened to a span of like we were saying a day. And if you just started with like if that could be your day and what does everything need to look like in order to deliver on that?

69:14 I think it changes the game for individual ICs. So first I'd say start by building. It's the Most biggest unlock. 2 as you're building it's kind of important to figure out what are the systems that you need in place in order for you to. It's easy to kind of build something and then say oh it doesn't work. Like I'm just going to. How many times that's happened to you where you're like it's not working. I'll just kind of scratch the idea and kind of let it sit.

69:46 I think the most curious and the most kind of curious of the PMs are typically this is where having a data layer like Arise and the observability platforms are really helpful is that you might not know why your agent gave you a bad response or why the outcome wasn't that great, what it was doing. And so getting observability to understand, kind of like we were talking about with just a simple example of the tennis players, how do they look at their plays and figure out what went wrong and how do they that one person better every single day if you could n plus one your output every single day.

70:29 I think that the story is no longer about observability. Oh looking at your data, the story is about self improvement and improvement of yourself as a pm, but also improvement for the products that you are building.

70:43 So we used Arise as open source sort of Phoenix platform and then we used Arise, the paid platform to do this. Those are two options. How does somebody make a decision what does the overall ecosystem look like and why would they choose Arise?

70:59 Yeah, great question. So Arise Phoenix, which was kind of the open source one that we pulled all the GitHub issues from today, is an amazing option if you cannot send your data to an external platform. And for most enterprises, most teams building any agents that have any PII data, it's just a reality is that they want to self host some initial observability so that they can get a feel and get started and get an unlock.

71:30 And so Arise Phoenix is. I think even Hamel's tweeted this before. His most favorite open source tool for observability is Arise feature Phoenix. It's got super permissive license, it's got almost everything that you just saw in the demo today out of the box for you. And all the skills that I shared using cloud code, all those skills exist for Phoenix too. You can just go open up, build an agent and say hey, help me instrument it, help me figure out insights from my traces, help me go write evals.

72:10 Phoenix will actually go and do all that for you today. Typically where teams start to feel the paid kind of platform, the enterprise platform kind of makes sense is when obviously data volume starts to scale. We have teams that send us just the volume of. I think it's a good thing is that these agents are starting to find product market fit in this environment Right now it's that LLMs, the models are getting better, products are starting to find product market fit and so we're starting to see almost like terabytes of data.

72:49 And so it is the volume and the scale is a big reason why for teams that need that as our agents start to get mature, it makes a ton of sense for you to kind of have a more scaled out platform for observability. This is where arise AX is kind of uniquely fit to solve that problem. We do this really well because we've actually had to invest in our own data store that we've been building for a while now, adb, and it's a data store that's designed for AI workloads from day one.

73:25 So let's say I'm figured out I need to pay for it. I have the huge amount of data. How do I decide who to work with?

73:32 The reason to pick a rise is really we're the open and independent, most independent platform out there. We are independent of framework. We don't actually care what framework you use. We have teams using everything from LangChain, the cloud, agents, SDK to teams that are building without a framework. And so we deeply, we're agnostic of whatever framework you, you use. The second thing is we deeply believe in the independence of your data. All of our trace data that we collect lives in open formats.

74:06 You can actually using our ADB data fabric, that data can be directly sent back to your data warehouse. And the reason that's really powerful is because you don't want your agent trace data, which is so valuable to be locked inside of a proprietary platform. We make it accessible so that you can actually use the agent trace data as part of your context graph. We're also independent of instrumentation. If you don't know, we're actually the inventors of open inference.

74:39 Our competitors, every single one of them that you mentioned, all use our instrumentation and they've actually linked to it in their docs. And so we actually own kind of, we built probably the richest telemetry and it kind of shows in the fact that our instrumentation is widely adopted in the ecosystem. And then the last one I think is just, I think we've been consistently one of the most innovative in the market. We were actually the first to shipping LLM as a judge.

75:11 If you go back to 2023, you look at Phoenix in the repo and you'll see kind of LLM as a judge. We were the first to release open inference instrumentation, Alex, that you saw in kind of the product. We were the first to actually have an agent built into our product. The skills that you're actually looking at that we kind of showed how you use all of those skills. We were actually the first to have and release them.

75:35 Hamel actually did a talk with Mikhail on this about it, our open source source lead. And then I was mentioning we have kind of the first and only way right now in market to actually take all of these agent traces and have them as standard formats as part of your context graft. And I think it just shows, you know, we are, we're probably the fastest innovator in the space right now.

76:03 What it, what are the things if somebody has just two hours this weekend that they should concretely go do and take away besides just they've watched this episode, but now they're going to actually make impact in their career.

76:17 If you, if you have any two hours this weekend, I would say literally what we just did right now, which is build, build an agent for yourself, whatever would take away a couple hours of your week every week. Like just something repetitive that you do every single week. And by the way, this isn't just for like, if you're someone who's in product marketing and you're writing release notes every week, like, what is just a workflow that you do every single week that takes a couple hours of your week.

76:49 Try to build an agent to go do that. And I think what you'll learn out of that is one, how insanely easy it is with Claude code. And then you'll also, on the other hand, realize how much work it takes to actually make it really good and so to make it get better. Past that initial kind of vibe code, the evals and the observability are so important. And so I said this in the beginning, but any product person that has used observability and is looking at their traces and looking at your evals, you're probably already in the top 1% of PMs in the world right now.

77:28 What are the biggest mistakes PMs are making when they do evals?

77:32 I think the biggest ones is not first not starting with actual trace data. I think if you're just starting with kind of what you think are problems, that's really hard. Like even the skills, for example, that we use today that Claude was using to build the evals, what's powerful about it is that it's actually we're trying to instill best practices. It's actually looking at all of the trace data to help and suggest what the right evals could be.

78:06 So I think PMs need to look at the evals don't just come out of magic, they come out of your they come out of trace.

78:14 All right everybody, I'm going to put up Arise's pricing page for you. This is how much Arise costs. Now here's the cool thing. If you want to get AX Pro for 12 months for free for your team because you're convinced you want to create self improving agents, you can do that with Aakash's bundle or you can just use the free options that she's talked about. Right now, Phoenix and AX Free to get started. It's that simple.

78:41 I highly recommend every AIPM master the AI eval skill. Arise is one of the easiest ways to do it. Aparna, thank you so much for lending your expertise.

78:52 Awesome. Thank you so much Akash and it was awesome to be here.

78:57 I hope you enjoyed that episode. Couple things you can do to Support the show 1 Comment 2 Review those ratings and reviews really help other people understand the value and the production that we are putting into this. That right? This wasn't an easy episode to produce. We put in a ton of pre work. We edited it for you. We brought in the best guests. If you don't mind sharing a rating and review, sharing the episode with others, making sure you are subscribed, that really helps the show do bigger and better productions.

79:26 I'll see you in the next episode. Here's one of those that YouTube thinks would be a great fit for you.