here's how i shipped 2,500 PRs last month to production - Lauren Tan https://x.com/poteto

0:01 Hi, my name is Lauren, you might know me as Potato on X. And I work on Grokbot at SpaceX AI. So last month I did something pretty crazy. I shipped 2000 pull requests to production. A lot of how I'm able to do this is through trust, and I think a lot about trust in terms of how I can trust my agents to produce high quality work even when I'm not there.

0:33 And my argument and thesis for this talk today is that if you set up your environment for your agents really, really well, you can end up with something that looks more like a personal or even team software factory, where you're producing very high quality code at much greater rates than before. But I'm not a fan of the term software factory. I like the analogy of a Michelin kitchen better, where, you know, as technologists, we're not really producing, we're not mass producing a product on an assembly line, but the work that we do looks very creative.

1:15 It's very. It's the act of building a product and it's art in some sense. So, you know, even though with agents, we're not cooking the individual components that go into the product anymore, we're still responsible for the final outcome and thinking about our kitchen setup. Right. Because depending on how you set up your line cooks, your sous chefs, you know, the kind of equipment they have, the kind of training they have, you know, dishwashers, and the ratio, I guess, of line cooks to dishwashers, all of these ingredients go into making the final product.

1:59 And I think this analogy is really apt. So I wanted to talk a little bit about a story before we begin, where six months ago, when I first joined Cursor, before we were a part of SpaceX AI, I obviously had no agent skills to use. I just joined the company. It was a fresh code base and a fresh product that I was working on. At the time, Cursor was building the replacement for the Cursor ide, which is the new Agents window.

2:39 Before I joined the Cursor Agent window had quite a lot of performance issues, and my manager at the time asked if I was able to help them. And since I had spent some time on the REACT team before I joined Cursor, it seemed like a good fit. But when I first started, I quickly realized how manual this process was. Obviously I have done performance work before, but at the rate at which pull requests were being landed, it was just, it felt almost an insurmountable wall of pull requests that just kept coming in.

3:17 And I had no idea whether or not the Performance of the app would be regressing. Right? So a lot of my early time on the Cursor team was spent looking at Chrome devtools and the performance and doing performance traces and taking heap snapshots. And it was extremely, extremely manual. It was so manual, it got to the point where I just got really frustrated and started to think about, wait, we have agents, what am I doing?

3:52 And so I started thinking about verification skills where what if my agent could actually run the application itself and take the traces for me, understand the traces and find the hotspots and basically hill climb to better performance in our application automatically. And throughout the last six months of being at cursor in SpaceX AI, you can really see that my productivity has skyrocketed and I never set out to ship 2,000 pull requests a month.

4:29 That was not a goal of mine at all. But, but I realized that all of the skills, all of the tools and code based changes I were making laddered up to this idea of trust. You know, I didn't know it at the time, but I had this thought in the back of my head which was, you know, I am the bottleneck and I need to be able to take all of the knowledge that I have as an engineer and impart them into, into my team of agents so that I didn't need to be the blocker for everything.

5:04 And you can clearly see that it's, it's paid off. So I think it really comes down to trust. But how exactly do you build that trust and where do you start if you are, you know, wherever you are in your journey of using agents? So when I started, obviously I was in this category, you know, the 1 to 1 to 5 range, where you still feel like you have to babysit every chat and conversation and you're just constantly course correcting, you know, you're intervening, you're correcting your agent so that it does the right thing.

5:51 And if you're not there, basically nothing gets, nothing happens and the agents do the wrong thing. And I would actually argue that this part, this phase of, you know, using agents is actually the hardest to get out of because it's not always very clear how exactly you get out of it. And again, it just comes down to trust. You know, the reason you're unable to go from one one or one to five agents to something more like 100 is because you don't have trust in your agent's work yet.

6:27 So if you don't have trust and you Try to spawn 100 sub agents or cloud agents, you're going to quickly find that you're just going to get a ton of slot pull requests and a bunch of regressions and a bunch of bugs shipped, and no one's going to be very happy with that. So that begs the question, how do you trust your agents more? So for me, it came. It really started when I, like I said, I meant I joined the Cursor team and I was starting to work on performance, where I realized the need for verification.

7:06 So when I say verification, there are sort of levels to that because, you know, on the, I guess the lower end of the scale for verification, you have things like the verification skills that I talk about, where you teach your agent how to run your application and use things like the Chrome DevTools protocol or whatever other protocol that you have for debugging and teach them how to debug the application, take performance traces, take heap snapshots and so on.

7:44 And then on the opposite end of the spectrum, which is much, much harder and still very much an open question, is more formal verification. Where, you know, maybe you rely on formal methods or, you know, languages like Lean or TLA to do, you know, verification so that you can check that, you know, your business level or business logic invariance are, you know, are always true and that you can formally verify that your application is always in a correct state.

8:25 But I would say that even if you don't have the ability to run formal methods, and very few people really do that, with verification skills, you can get very far. When I joined Cursor and started to work on the Cursor agent window, the first skill that I built was a skill called Control Glass, which is a verification skill that teaches the agent how to run the application and take traces, like I mentioned, and it does that through the Chrome devtool protocol.

9:01 Something interesting about that skill is, and I kind of iterated my way to this, it didn't start out this way, but the control or verification skill really has two components to it. The first part is obviously a cli. So you want your agents to be able to reproducibly be able to run the application and collect traces and collect evidence that empirical evidence that code is working, that your performance bar is being met.

9:41 And rather than have your agents create scripts every time that can differ between agent sessions, you can actually create a CLI that's within the skill directory. And then your agents will just use that every time. And then, of course, you need to actually invest in it and make it good so that it can handle all sorts of different use cases and be able to run the application correctly.

10:16 Another thing that's also really important is this idea of a feature map. So a feature map really is something that I kind of coined, I guess, where we started using these control skills within Cursor, then we quickly realized that a Slack report would come in and a user would post a very vague screenshot, like a very small slice of the UI and just like three question marks. And the agents using our control control skills is just no idea.

10:48 Like it could run the application, but it would just be guessing at what exactly the user meant. And so I had this idea to create something called a feature map, which is, I guess, kind of inspired by like a sitemap. It essentially is a form of materialized memory. You know, like how exactly does your application work? What features does it have, how does a user reach it in terms of keyboard shortcuts or what DOM elements to click on, things like that, and what all the different features do.

11:28 This feature map is stored in the skill itself, in the code base as part of the skill directory. And we have an automation that maintains this feature map as well. When we combine the CLI and the feature map, we quickly realized that this combination was very, very powerful because now agents can not only reproducibly control the application and take traces, but it could also understand requests that came in from internal users as well as external users.

12:02 And so we very quickly realized that the control verification skills were so useful that they've become more or less critical infrastructure for our team, and we constantly maintain it. But the ability for an agent to verify its own work is extremely powerful. And we have spent a lot of time on this skill and it's very, very powerful for building that trust.

12:34 In addition to verification, of course, you want. Because verification is really about correctness, Correctness to me is really about does the thing, does the feature or code do the thing that you want it to do? Right? Like, does the, you know, the checkout button, does it actually check out the card? Verification is really important for that because you, you get empirical evidence that this feature actually works, but it doesn't tell you much about the performance or the code quality of that feature.

13:11 And that's where you start thinking about skills that teach agents to work like real software engineers. So I've built a plugin called pstack. I'm not going to talk about the plugin too much today, but a lot of inspiration for that plugin, which is a collection of skills that I've created, are inspired by the kind of workflows that I personally have used as my, in my time doing software engineering for all sorts of different types of tasks.

13:46 So debugging, you know, feature development, prototyping. There's a whole bunch of different playbooks and skills that should be in PStack that teach your agents how to write code the way that you want them to. And you know, this is where, you know, the more experienced engineers on your team can really contribute to set up a team repository of skills that just make your agents a lot smarter.

14:18 And when you combine those skills with verification skills, then you, you're able to get the point where your agents are able to not just verif verify that the work that they're doing is correct, but that it's also high quality. And again, because of verification, you can collect real performance metrics, you can get real numbers and statistics and telemetry on the performance of the application.

14:50 So I think that's a really important part to invest in. Another thing I think is really important is refactoring and rewriting your architecture to be more agent friendly. And I almost want to say that this is one of the most important things you can do as a software engineering team, because if you really, truly believe that agents are going to be writing all the code in the future, then we need to design our code bases so that they do the right thing by default.

15:25 And you'll find that there's this, I almost want to say, like a scale or continuum between these different pieces of building trust in your agents, where I have like five of these points here where the code base is really like the best form of memory because agents love to extend existing patterns that they see.

15:55 And you know, I think this is just the nature of how LLMs work, where they're more likely to use whatever it is in their context window to make changes. And of course, the files that the agents read and opens are part of its context window and therefore the code base is a really important part of that because, you know, agents aren't going to just refactor your code in every single pr, they're going to just look at what's already there and just extend it.

16:28 The next level, I think, is about static analysis, where you have linters, you have compiler diagnostics, you have continuous integration, and these are guidelines and constraints that you can enforce in your code base so that whenever you correct your agent, you find that, you know, they just keep making the same mistake. You can add those as lint rules, or even better, you can refactor your code base so that the mistake that the agent is making becomes categorically impossible.

17:05 And then a step above that, right where, where. And this is where we're starting to get more into less of like a hard constraint and enforcement and more into the realm of guidance. You have things like rules, you have bug bot, you have skills, which, you know, your agents will obviously sometimes and mostly use when they're doing their work. But there's also a chance that it might, for various reasons, you know, forget to read a rule, or maybe the user that is piloting the agent ignores them.

17:40 So these aren't quite as enforceable, but also an important part of setting up your environment so that you can really trust what your agents are doing. And then finally you have the style guide, which is really only enforceable by humans in a code review. I guess you could put these in your rules and bug bot and skills as well, but if you don't, then you have this big glaring hole in your review process where now humans have to, you know, look at every single line that's being changed and remember to comment.

18:17 And with the rate of pull requests that are coming in, it just, it just becomes impossible. So I definitely wouldn't recommend, you know, relying only on the style guide. I think the style guide or, you know, like looking at human reviews is a good place to start in terms of what's missing. But you should really invest the time to think about the other four parts. Your code base, making things categorically impossible through better data structures or algorithms, static analysis.

18:50 And then of course, you layer that with rules and bug bot and skills. On the code base front. In the Grokbot code base, we actually have invested into setting something up that we call Dune, which is our agent friendly framework. The inspiration for Dune really came about from a lot of performance issues we were seeing in the cursor agent window.

19:23 A lot of lessons came out of that exploration. But the key principle that we landed on is really that agents love taking shortcuts. So what if we designed a framework such that the shortcut, you know, the easy path is the right path for agents, and also one that would be a code base that is, you know, maybe pretty annoying for humans to work in because it's so locked down in terms of what you can do and what you can't do, but it actually creates the perfect environment for agents, especially ones that have very minimal context, because not every contributor to your code base is going to be an engineer anymore.

20:07 You can have designers, you can have product managers, you can have CEOs going into the code base. And shipping features. We want to really think a lot about how we invest and set up our code bases so that even agents that are piloted by busy people with not a lot of context can do a good job by default. And like I mentioned before, your code base is really a form of memory for agents because they love to extend the existing patterns that they see.

20:43 And the reverse is actually also true, right? You can invest the time to set up your code base in a way that things are, you know, bad patterns are categorically impossible or you have like lit rules that prevent them. But the reverse is also true in the sense that if you have existing anti patterns, you will actually find that these will spread. Kind of like a virus where you have like one small workaround or a comment that explains a workaround and you'll quickly find that agents just love to copy that.

21:19 And then in a matter of a few days or a few weeks, you'll find that the workaround has spread everywhere and it's now becomes, it has become like a de facto pattern for all agents. And that's a really, really bad place to be, to be in. And that goes back to what I was saying about why it's really important, you know, whenever you're correcting your agents, that you invest the time into thinking about code based changes and static analysis and of course layering them with rules, good rules and bug bot and skills because of that reason.

21:59 And another analogy that I like in addition to the Michelin kitchen is this idea that your code base is kind of like a garden where you have workarounds that seem kind of innocent at first, but then because of the nature of agents, you just copy that pattern over and over again and you quickly end up with a very vibe coded code base that is a pain in the butt to maintain and has a lot of performance issues.

22:32 So in my opinion, the perfect agent code base is one that's so locked down that again, it's really annoying for humans to write code in, but it's so conventional, it's so standardized that even innocent looking patterns are just forbidden. And the best example of this I have is actually something that seems very, very innocent when you look at it, but when you think about it, it's actually really bad. And that pattern is agents leaving comments in the code.

23:08 Now, you know, when I first saw agents starting to do this, I initially wasn't. Well, I did think that a lot of them were slop. But I also thought that, you know, it actually doesn't, it's not a bad thing, right? I guess if agents are leaving comments in the code, because as humans, we left comments in the code whenever we saw, you know, edge cases or we needed to actually make a workaround or, you know, leave a note to ourselves or a colleague on a particularly tricky part of the code base.

23:41 But what I quickly realized when we saw this happening in the cursor code base was that agents were just using the comments around the code as justification for why it wasn't going to solve the actual problem and instead paper over it with a band aid or a short term solution. So in Dune, which is again the framework that powers graphbot, we made the choice to, we made the choice to actually ban comments for that reason so that agents would not just copy that pattern and propagate it everywhere in the code base.

24:28 So my pitch here is that every team really needs something that, you know, a role that I'm calling a gardener in the same way that, you know, with a real garden, you need someone who is thinking a lot about the, you know, things that can kind of creep in and grow in ways that you don't want. Like, you know, you have weeds, you have, you know, just other types of organic growth. I don't really know gardening that well, but you have things that, you know, unwanted pests and stuff like that that kind of creep in empirical based and so you want to nip them in the bud as soon as possible before they start propagating everywhere.

25:15 A lot of the principles behind Dune are really centered around these three things. First of all, we want to delete tech depth that we already have for reasons I just mentioned. We want to keep or enforce a single paved path for most blessed patterns. You know, there should be one conventional way to do something so that agents don't really need to guess and there should be enough guidance in the code base, in CI, in lint rules so that the agents are guided to do to follow that path.

25:53 And finally, whenever you see tech depth or bad patterns, your instinct should be, I need to write a lint roll against it. You don't always have to clean it up immediately because if you write a lint roll, you can at least stop the bleeding and which, you know, doesn't solve the problem entirely, but it at least prevents it from growing. So I definitely recommend, you know, really thinking a lot about how you can guard against anti patterns so that they don't spread like a virus and then also spend the time to actually, you know, get your agents to clean them up so that your code base is just constantly kept in a state where you would be happy if an agent were to copy it.

26:38 That's the kind of mindset that I would recommend having. And then I won't actually go through all the details of Dune itself, but I'll just kind of gloss through some interesting parts. So again, as a reminder, Dune is the architecture, the client framework that we built to power Grokbot. We've invested a lot into, you know, all the things I was saying where we have conventions, we have a lot of conventions about where code should live and where and how code should be imported between them.

27:11 So in Dune applications, you know, there's different concepts where, like, for example, features are all co located in a single folder. You have an entry point point that's in the react part of the code that determines. You can kind of think of it like a route. You have transcript cards that show up in the graphbot application, you have a host that runs on the graphbot virtual machine. And then of course, you have your client which powers the overall application.

27:44 And we have a lot of strict boundaries between these things, where, just as an example, things that run on the main process or the main thread in Electron aren't allowed to be run on the renderer thread. And we keep that separation very intentionally because of lessons we learned from Cursor's Agent window, where we would sometimes see code accidentally get imported into the renderer thread and slow code.

28:18 And since on the render thread you want your UI to be very smooth and performant, you need to make sure that you don't have any long tasks or, you know, things that take longer than 16 milliseconds for if you want like 60 frames per second or 8 milliseconds if you want 120 frames per second. And so your renderer has to be constantly in a state where you can really kind of chunk up the work and not do them all at once.

28:53 And so we have code within Dune that enforces these boundaries through the import and dependency graph. But yeah, that's just an example of a pattern that we saw lead to really bad performance that we categorically eliminated through the architecture of Dune. And then all of these other pieces aren't that interesting. But again, the core theme here is not about Dune, but the idea that an agent friendly framework of your own is actually very, very powerful and you can encode all of the learning set in you and your, you know, your best engineers on your team have tribal knowledge of.

29:45 And I think the lesson here is that how do you take that away from, you know, what used to be in the style guide process of reviewing code and, you know, engineers reviewing other engineers work and leaving comments to extract, almost like extracting that knowledge and encoding that into the framework, into the code base itself so that the code base access the memory, right? It's the thing coming back to this idea that the code base is just the thing that it's the materialized snapshot of the state in which you want your agents to extend, and you want that code base to be so pristine, so great that the next agent that comes along is just very likely to continue that pattern and keep it really, really good.

30:38 And if you spend enough time on this process, like I mentioned, you can really set up a Michelin kitchen or a software factory where, because you've spent so much time on all of these pieces that allow you to trust your agents, whether it's in the code base, whether it's lint rules, whether it is, you know, diagnostics or rules or bugbot or skills, these layers come together and provide you a lot of trust.

31:14 Because now, you know, just imagine for a moment you're working in the Rockbot code base. It's super locked down. You know, it's like almost impossible to write bad code. So, you know, you can even an agent with very little context, you know, even an agent with not a lot of reasoning can come in and actually write code that's good. And going back to my example about the Michelin factory, I think there's a lot here, right, where we're setting up our agents, our bots with skills and tools, we're training them, we're setting up our kitchen in a way that makes sense for the agents and bots to do the right thing by default.

31:59 Whenever we see, for example, in the kitchen example, if we notice that one of our cooks or dishwashers is constantly tripping over something, of course we need to fix that, right? We need to problem solve and ensure that, you know, others don't trip as well. Because, you know, in a kitchen, it's a very dangerous place and you don't want to hurt yourself. It's the same mindset I think, that we should have with our code bases. How do we set it up so that even agents without a lot of knowledge can and do a good job?

32:34 And I think with, you know, Grokbot, Grobot and Cursor play an interesting role together, where Grobot is really great at providing what I call the outer loop, because you can connect Groffbot to lots of different, you know, different connectors, like Slack to Datadog, Sentry planetscale, whatever services that you use, and you can aggregate all of that information together and use that to make really good decisions for itself.

33:07 Some people call this like a company brain. I don't really think, I personally don't think you need anything that sophisticated here because agents are really good at using tools. And so if you connect these tools through Grokbot and you start having your Grokbots auto kick off things like cloud agents, you can actually find that it's really not. You don't really have to invest in a lot of infrastructure to build a software factory.

33:38 In fact, I'm gonna, you know, cross, cross out this, this term because I don't like this term. I think you can set up this personal Michelin kitchen for yourself through Grokbot. Things like Grokbot routines which let you subscribe to, you know, slack threads, to sentry alerts that let you kick off things automatically. And when you combine all of these things that I've been mentioning, you know, your code base, your rules, your skills, they all compound.

34:10 And Grokbot will be able to, you know, automatically respond to events that come from the outer loop and then kick off cloud agents. And you can also set up cursor automations and use our SDK to set up additional bots as well that reuse a lot of these pieces of agent infra that you've set up and allow them to do much more complicated tasks.

34:42 So if you, if you, if you've done all this, then I think you can get to a point where, you know, I have some screenshots here of some of our automations and our agents in that work on cursor where we are automatically reproducing bug reports, we're automatically opening pull requests, we are essentially adding a lot of value to the entire team because all of these things compound.

35:15 So if we kind of zoom out again and go back to this graph, I think that to kind of close off the talk, if you spend a lot of time thinking about all of the pieces that you need to be able to ascend the trust graph, you start getting to a place where you can really trust your agents more and parallelize your work and also empower your entire team to build on top of these pieces of infrastructure for your agents and empower everyone, every engineer on your team, every builder to be extremely productive and be able to write high quality code.

36:01 So the last thing I want to leave you with is actually this piece. Sorry, not that piece, but this piece. I think if there's only one thing you take away from my talk. It should be this slide here, which is these are the activities that will help you build up towards a high trust environment. You know, whenever you find yourself correcting and interviewing your agent, you really want to think about it from these five pieces and where is the most effective step in this sequence in order to make your agent, you know, much more trustworthy?

36:45 And of course, I definitely recommend thinking about thinking about it in this order, where you either invest the time to make that pattern categorically impossible through your code base and architecture and data structures, or you start looking at things like static analysis and then you layer that on with rules and bug bot and skills. If you do all of that, and you also spend some time thinking about your code quality in terms of skills, you get to a place where you trust the environment so much that your agents can just be free.

37:27 Right? And personally, I have spent a lot of time for this, for Grofbot's code base, for example, and this is really the secret. Well, it's not really a secret. It's a lot of hard work. But I hope you found this talk useful and please reach out to me on X. My handle is potato with an E. And I hope that you'll have a lot of fun and successfully your own Michelin kitchen.

37:59 Thanks for watching.