transcribe

How Ramp engineers work with AI agents at every step

Claude · 21m · transcribed 5d ago
More from Claude Business
𝕏 Share ▶ YouTube 📥 PDF 🤖 .md

Transcript

0:00 I told Fable to fix all our import cycles. I also told it to make our app lazy. So the app boots up, and it’s So the app boots up, and it’s an enormous amount of Python modules and Fable made a lot of progress in both of these. A lot of this code was merged. Really understanding what the boundary of where these models break is very important, because those sets of problems are the problems that we would want our people to try when the next Fable comes out or the next release comes out.

0:32 One of the first things I tried when I got my hands on Fable was I wanted to find a use case in Ramp that we can empirically verify. So it must be a large piece of code that I haven't read, that Fable wrote. It must be in a non-product place. So something in CI. So this is for us in our testing suite. And I wanted to empirically verify it. To be able to produce a lot of data and check if the code is acting like it should.

0:56 It's been working really well. We're running it for a few weeks in shadow, and it's been consistently faster than what our current implementation is. But I threw the deep critical problems in our codebase. So we have this large monolithic Python code base, and I told Fable to fix all our import cycles. I also told it to make our app lazy. So the app boots up, and it’s an enormous amount of Python modules and Fable made a lot of progress in both of these.

1:21 A lot of this code was merged. Really understanding what the boundary of where these models break is very important, because those sorts of problems are, the problems that we would want our people to try when the next Fable comes out or the next release comes up. I have these tests that I give every model when they come out, and I just sort of, you know, no model ever just was able to do all of these things.

1:40 But this is the first model that just did all the things. And when Fable struggled, I just used a dynamic workflow, you guys have used it. This is like where Claude has a bunch of sub agents that it orchestrates. And using this sort of like algebra in the sandbox. But essentially it's like a new form of test time compute. And so you just tell Claude, you know, use a workflow and Fable does it. Just yesterday, it actually reduced our CI time from, I think, 18 minute P50 to 6 minute P50.

2:08 Wow. And it just like optimization after optimization after optimization. And they just kept profiling it. The code landed then it waited a day and used a routine to schedule itself to run a day later to get that real production data. And then it just like repeated this for days on end until it landed all these wins. And then it showed me a chart when it was done. Rahul loves that. He’s a huge CI time minimization advocate.

2:31 It's all I think about. I just have a follow up question for you. So when you're doing these, dynamic workflows, I would have to expect that you've already built up the familiarity with Fable as a main agent? Like a foreground agent, right? Because there's like a lot of learning from what the 40 or 100 different agents are doing in the background that maybe you're not seeing, but you do have to have comfort by that point because that's a big workload.

2:59 Because the agents are, you know, able orchestrate, so the agents might have like work translation they might not, they're executed, you know, in parallel or serial. You might add more rounds of like adversarial verification or whatever it feels this task needs. So yeah, I just trust it. Essentially my mental model for dynamic workflows is the test time compute. It goes from like low to medium to high to extra high to max. And essentially the way to think about this is this is the maximum amount of thinking the model can use.

3:26 It won't always use a, but it's the maximum. That's why you have it on the thinking dial at the very end, right? Exactly. And for me, dynamic workflows are just like it's a next level. It's like another form of test time compute. It's new. How do you think about loops versus dynamic workflows for achieving some sort of long horizon thing? So loops are kind of like repetitive work. And dynamic workflows are like dynamic work. Like you don't you don't exactly know what the steps are ahead of time.

3:51 Like, I use a loop, for example, for babysitting my pull requests, to fix the CI, and rebase them automatically. But then I use dynamic workflows for things like system optimization, where you don't necessarily know what the next optimization is. It's a total mental model shift. It does feel like using Claude Code for the first time, where you have to start relinquishing a lot of the software engineering workflow, including running commands to this agent, right? I sort of feel like for me loops is, if you have a bunch of engineers doing work, loops are kind of slicing a horizontal off of it.

4:23 Like if there's one task every engineer does every day, you can maybe take that and put it in a loop or in a routine. And this is something like a code review, babysitting a PR, addressing feedback. You know, like we just have dozens of these. Like, I have one, for example, for deleting dead code. This is a routine that runs every day. And then on the flip side, you can do this vertical slice. And for us this is like Claude tag.

4:46 And you know an example of that is I'll have tag ship and experiment. It'll make the experiment, it'll land the PR, and then it'll set a reminder for itself, using a routine to monitor and check in the next day, it'll make sure that the exposures are balanced. It'll crank up the exposure, make sure the experiment’s running, and maybe a couple weeks later, it'll be like, all right, I'm going to ship this variant and it puts up another PR for that.

5:07 And I wasn't in the loop at all. Like at the beginning I asked Claude to do this. I stamped the pull request, but the rest was just Claude. Okay, I want to like detour a little bit. I want to hear about your coding setups. So iTerm2, pretty vanilla. No IDE these days. And as many pains as my monitors can handle. Pretty barebones Claude Code set ups. so not a lot of plug ins or skills or MCPs, pretty simple CLAUDE.MD, inspired by your, you know, your Twitter posts for the vanilla setup.

5:41 I think it's the best way to learn the models. And then good amount of subagent use, adversarial review. Yeah, mine’s gotten increasingly background heavy, and almost most of my sessions are information gathering. It's like, why is the memory spiking on the service? Or how can we get this project done faster? And it's a great way to like, fan out a lot of sessions and gather a lot of context. When I do local things local with Claude Code, it's usually maybe more hands on programing where I, it's closer debugging, or I need more services or context on my computer.

6:14 Has that changed a lot over time? Like do you start with a sort of, you know, like an Austin set up of like just terminal, terminal, terminal, terminal. And then you kind of move to this? I mean, we have so many services, a lot going on databases and message queues and Reddits and all that. And so having multiple instances of local dev running can become a constraint very quickly. Especially with the latest models, I think they require much less hand-holding.

6:37 And sometimes you just gotta let them cook, get out of the way. And so I found myself carrying my laptop with the lid open a little bit too much. And so then we decided to move. That’s funny. I know exactly what you mean. Yeah, make sure Caffeinate is running, right? So we've implemented agents at pretty much every part of our business, but especially in the engineering lifecycle. So if you take the process of building and shipping software, everything from coming up with ideas, figuring out where the bugs are, getting notified when there's problems in our logs and our systems, to writing the code, to reviewing them.

7:09 And sometimes when they're after deployed, looking for how they're doing in production and seeing if they're doing the thing you want. We've tried to build systems along this whole stack. We've also thought about it from the lens of security, trying to find bugs and other issues. Okay, so now you're at the point where you're using the model kind of everywhere throughout like the whole lifecycle. How did you get there? What was like, what was the first place where you started using Claude Code?

7:34 And then how did agents kind of expand out of that? We were seeing and slowly realized, clearly this thing's going to continue to improve and maybe we shouldn’t build for 2.7, maybe we should build for whatever is coming next or the model after that. And over time as we built these harnesses, we’ve learned to step back and just wait it out because a lot of time we end up removing this scaffolding over and over again because the model is just like outgrown harness.

8:01 At any given point, when there's a shortcoming with the harness or the model, we've tried and we're not perfect, we also need to make the product work today because otherwise we won’t have business, but we've tried our best to go the other direction and give the model more tools, more context, more agency with the goal of almost, being able to treat our agents like a coworker. So, hey, can you go figure this out? Like there seems to be some sort of exception that's popping up.

8:27 Or maybe this customer is complaining of a certain issue. And we want the models to be able to access the right systems, the right level of access, and produce the right amount of right code. And so just wanting that simple goal allows us to figure out what we need to do to give the model enough access to do these things. I think it's a velocity bet in a lot of ways. Right? Because you're basically saying, I think the stuff we would put in place to make this work now really well is going to become technical debt really quickly.

9:01 And that's going to slow us down. If we aim a little further in the future or sometimes a lot further in the future, you know, we'll actually make it further with the resources we have. I have so many questions. But maybe like one direction we can take is, how do you make sure they have the right guardrails? Like, you know, like they can access this data, but not this data, or how do you make sure the cost is under control?

9:24 How do you make sure the code quality is good? And how have you guys thought about this, like as you scale up the systems? Yeah, so we've also, at various levels on the stack, tried to implement safeguards. We also studied the trace a lot. One of the things that I think we've tried to focus more on is studying individual traces and less on aggregate level benchmarks. Benchmarks do give us a lot of information cross model, but a lot of the time there's usually a correct trace.

9:48 It's like, what is the command the model should have run in this scenario, and why did it not get there? Is this a context issue? Maybe it does not have access to the right tool? And just following these simple traces for workflows that should work allows us to like get there in the right way. We've implemented a lot of layers of defense across, I mean we’ll continue to do that. It's how, how many layers we have also allows us to move faster and give it more agency and more access.

10:12 And so again, at every part of the stack, we've done everything we can to give the model what it needs, but nothing more. So you're essentially like, you go to BigQuery or Data Dog or whatever. And you give it like a read only service key. This is essentially how you think about it? That's right. Yeah, exactly. So a lot of the time so let's say that, you, I just want to be able to say like, talk to my agent, like I talk to my coworker.

10:31 And so we're almost focusing on the default experience, the iPhone experience. Where you open it up, there's a text box. You just say what you need to get done, not how to do it. The prompts must be declarative. So we're not we don't want people to instruct the agent to do it in a certain way. We just want people to say, implement this feature, or fix this bug, or help this person out. And over time, especially when you focus on the correct trace.

10:57 So what must the agent do? It must first query the source and then it must query these other sources and read the code in this in these repos. Just by focusing on what the correct trace in your head is you can then shape the agent trace purely through prompts and tools and skills to get there. And thankfully we're also on this exponential increase in model capabilities. So maybe if it's not working right now, you just got to trust that it will get there.

11:22 And just with that belief alone, just ship it and just wait. And the one thing I'll add is good old fashioned hard controls on top of that, like you said, like principle of least privilege stuff. The basics of just not even giving it the opportunity to be able to do certain things. And how do you think about like enforcing it? Is it like the security team's job to do this or are you like federating out the design of these sort of systems?

11:46 How do you think about that? The really exciting part about this is the infrastructure has been built by the security team, and it's been, the security team is very closely related to this. So they helped us set up the network access policies. They helped us get the keys. And they're also regular users of these agents. How do you think about cost controls? How do you think about code quality? What else do you think about as you scale it?

12:07 We're continuing to find and look for cases where, again, we can guarantee we know for sure the worst thing that could happen if, for example, this code has a bug or something like that, the effects are extremely constrained and we do have an upside and we're finding more problems like that. And we're trying to use this hammer for that. We're also expecting a massive increase in the amount of productivity, especially with the next few models coming.

12:30 And so we're readying our verification loops especially with CI and CD. I think also changing what our reviewers look out over time because as the models get smarter, they stop making certain classes of mistakes. And so you, it's not worth spending your reviewer tokens on that anymore, right? Yeah. I mean, we've invested in our own code review bot as well, which is also built on Inspect or our background agents API. We pull from some memories of things that we especially want to look for.

12:57 We have certain teams that write their own, skill files that look for certain things so that they can codify the knowledge that they have built up over the years into these files that allow people to move a little bit faster. It sounds like it's not like Austin and Rahul that are going in and just like breaking down every bottleneck. Although I'm sure you're doing a lot of this, like how do you create a culture where engineers feel empowered and have like the visibility and the tools, whatever you need, to find the bottleneck and to break it down?

13:23 It's just Ramp, right? Yeah. I mean, I think a lot of it is the culture that the company has built is a culture of experimentation, a culture of like building something that maybe didn't pan out, and that's okay. We've tried something. You move quickly. I think one of the things that has been helpful is, because we've had free access to all the tools, to all our engineers, we don't really like to impose a certain token budget or a tool budget, or tell people that they should use this thing or that thing.

13:49 And in general, it becomes a lot easier to speak the same language. I mean, it sounds like you guys just built like a huge number of these, like background agents, like various CPIs and systems internally. So you mentioned Project Glass, you mentioned Inspect, walk me through these. What are these tools? How do you use them? How are they built? Yeah. So Glass is where our, it's the home base for our non-technical folks. It's where they interact with the coding agent on a daily basis.

14:15 And it's been our belief since, since the beginning that everybody should have access to this power. And this velocity increaser. But you got to meet people where they are, you know, they don't want to be looking at code. All the technical detail is not going to help them go faster. And some things need to be set up ahead of time. Yeah. Inspect, at this point, is basically a digital coworker. We've tried to give Inspect all the tools that a Ramp builder, so product engineer design person, would have.

14:45 So this includes access to GitHub, and Linear, and Slack, and Datadog, and Sentry, and various other tools. And at this point you can ask Inspect to solve a support ticket, or fix a GitHub issue, or look at a Sentry error, or Linear ticket or Zendesk ticket, whatever it may be. It runs on Modal in the background. Your access it via web. Yeah. And a lot of people kick off stuff from Slack. So if you're in a conversation with someone about something and you @Inspect, can you go handle this or can you put up a PR to fix this or investigate this?

15:17 And that actually ended up being the main way that adoption was spread, because you would hop into someone else's thread and @Inspect. Can you help them with this? And they’d see it and go, oh, you can just do that? Oh. Great. Yeah. And every PR now comes with its own VM and it’s like running for a little while so people can take over sessions, collaborate, it's all link based. It's all multiplayer. It just works out of the box.

15:41 And again we've tried to focus on the correct trace, like, what should this agent have done? And try to shape it, to that way, so that it can do a lot. At any given point, sometimes people feel the urge to move back to local dev. We haven’t fully finished this project, but we've tried to give Inspect that additional tool, that additional repo or dependency that allows people to stay a little bit further in the background. I also do want to mention on call assistant.

16:08 So on call assistant has been running on and always ran on Claude Code. And that's another instance of just taking what works really well locally, proving it out locally, building up the skills and MCPs and prompts that make, essentially like an AI SRE run really well on incidents to root cause them and put up PRs of fixes and then just packaging that and having it run in a container with safeguards and guardrails.

16:38 So on call assistant runs on every instant that gets assigned to our engineers. So that includes customer tickets, customer support tickets that require an engineer, but also includes system level incidents. We're working on it. And then comes back in with a really solid root cause analysis in the Slack channel that we have for every incident. And then the incident responders interact with it. And we've had that running since late February or March. The stuff you can build on the primitives on this sort of Unix philosophy, Claude Code executable, it's just wild.

17:14 Yeah. We have a bunch of really similar tools internally. And, now Claude tag, which sounds pretty similar to Inspect. It sounds like. In a lot of ways. Yeah, it's also multiplayer. It's also kind of proactive. You know, it's in Slack. It’s sort of taken over a lot of these special purpose bots. I wonder if you guys are seeing the same thing? Yeah, we're seeing something similar. So more Inspect sessions are coming from automations than humans at this point.

17:38 So every time something is, So every time something is, there is some sort of trigger. Sometimes they're scheduled at a certain time of the day, sometimes they're from other external systems. Then a session kicks off and sometimes notifies people and channels or by DM. And I guess for this one also organizationally, culturally, how do you do it? Is it like each of these automations is built by different teams that's closest to it? Or do you have like a central dev infra or like AI team that is responsible for all of these?

18:02 It’s been very, surprisingly, very decentralized, and we're very happy about that. I mean, there are teams that maintain certain abstractions, in the Inspect team, as you mentioned, the Inspect abstraction, that is a bedrock for a lot of these automations. And if you let everybody build what they would like to build we’re okay with that. And we want that, we want more of it. All we can do is build a great product, so other teams are incentivized to build on top of us.

18:26 It's also a mix of, like, desire paths of people wanting the same, expressing the want for the same sort of thing, or building the same thing separately, and then the sort of platform team going, okay, let's make a solid thing for this. And vision from the platform team of we're going to need this when the model gets smarter. Yeah. I mean, so taking a step back, one of the things that we've tried to do is not impose limits on how much, how many tokens or dollars each individual spends.

18:55 We want them to be able to access any level of intelligence without limits. So because of that, we've tried to do everything else in our power to make sure that people can step up and get that intelligence where they want. So that includes things like defaults. It's using batch and flex APIs. It's using cheaper models for automations when they're not human controlled. So we always expect to stay on the latest frontier. And so we don't want features or people overfit on a certain model's behavior.

19:21 And then there's a good amount of just talking to people too. I think you and I have both done this where we see someone suddenly become a top spender on a certain month, like way above what they normally do. And we reach out to them and say, hey, what are you working on? You know, it looks like you're spending a lot. I'm curious. And if it's something that you're not planning on platformizing, but is platformizable let’s work together, let's do that, let's expand the impact.

19:49 And if it's a mistake then I'll help you with that. And then we can work on getting the costs down later if it's something you do want a platformize. So essentially it's like this, this culture of, like, experimentation and innovation. It's letting you just totally automate, like a big swaths of work that used to be manual before. So obviously it works. And so then kind of your job is to support people and optimize the use case after it takes off.

20:16 Yeah. And so the other way to look at it almost is like if you are in the positive ROI section where you know that every dollar you spend on tokens, you're actually making more than $1, you actually don't want to be minimizing costs anymore. We also expect the level of intelligence that fable has, the cost of that to decrease over time as it has for the last few years. It's not anything new. And we rather have, everybody at Ramp be familiar and really good at pushing the frontier and pushing with intelligence, making it sweat on hard problems sooner than later.

20:48 What is your advice to your peers, to other CTOs that are trying to figure out, what do you do? How do you adopt agents? How do you make your way through this thing that's happening in the industry? We've made a lot of progress in the models today. We have great tools at our disposal. But I think the thing that people don't pay as much attention to is also the rate of change and how much things are changing, over the last few years.

21:13 And if you pay more attention to that as opposed to the current snapshot, then you begin to see the pattern of like rising intelligence and agency, the ability for models to do more things. And I think we've tried to build for what comes 3 to 6 months later down the line, because sometimes when you're playing catch up and you’re building for what's available today, it might already be too late by the time you ship. And so, paying attention to the scaling itself has been very helpful for us.

21:43 All right. So with that, Austin, Rahul, thank you guys so much for taking the time and for hosting us in this beautiful space. Thank you guys so much. Thank you.

© transcribe · For agents Built with care and craft by Gokul Rajaram