transcribe

How Salesforce uses Claude Code

Claude · 16m · transcribed 5d ago
More from Claude Business
𝕏 Share ▶ YouTube 📥 PDF 🤖 .md

Transcript

0:00 Every new application or UX you're seeing in Dreamforce all probably had some level of Claude influence there. What are some examples of products or features that the team shipped that a year ago, maybe at last Dreamforce, just wouldn't have been possible before Claude Code?

0:31 I would say many. The whole Claudeforce was possible because of headless. As you know, Salesforce has been around for decades. And so not everything has APIs. So we tried to make it all headless. And it would have been a six month or a year long project to make all our applications headless, going through every single set up page and looking at the APIs. What's the gaps? We were able to do it in a matter of weeks and few months.

0:57 And now we are at like 80 - 90% of API coverage for all the clouds. In a matter of less than a few months with no meetings across the teams. I think that was one of those wow moments. Where we’re like, imagine this is not new code, this is preexisting code, for decades to be able to go do that. I think it was pretty amazing. And so on and so forth. I mean, there are so many such examples.

1:23 Every new application or UX you're seeing in Dreamforce, all probably had some level of Claude influence there. Is this how you guys think about ROI? I'm hearing you have a project and now you can build it much faster. You have a project and now it takes less people. So your team just has way more throughput. How else are you guys thinking about ROI? Yeah, from coding perspective, this was a tough thing. We were debating what does that ROI mean?

1:50 How do I know it's better? More? It can't just be lines of code because then people, you can just keep adding lots of comments. We work with Stanford researchers to create this thing called an effective output. It's a score that tries to say, what is this code that's being generated? How good is it? How effective is it? So it will give a low score if it's just making some simple changes or adding comments or a higher score, if it's actually changing logic or creating new things.

2:16 And that has been extremely helpful for us because we are able to look at, okay, has that changed using AI? And also then dollar per effective output, we're able to actually see measurably, okay, what are we doing? Are we moving the needle? And that's grown 2x for us. Which has been pretty substantial. And now we're trying to figure out what's the right metric for all the other functions. How do we measure it? Ok, they got better.

2:42 What does that better mean? Did they close more things? Did they have more sort of case volume they were able to handle? Every function we're trying to go look at that metric that would make sense for that function. And then doing the before and the after to see the ROI. The before and after, kind of introducing agents or making some change to the workflow? Making changes to the org. And then we also know what they're using, how much they're consuming from token perspective.

3:08 And then we're looking at the before and the after to see okay, is there a material difference? If not, why? Right. Is it because you're using it wrong? Or is it just not the right tool for the job? What is it? Walk me through how Salesforce uses Claude Code and how you've scaled it up over the last year. We use pretty much Claude Code for all our PLC, I mean, SDLC sort of life cycles. People did a lot of stuff.

3:32 And then we started bringing in more, structure around it. One of the big things we did working with you folks as well, is to really create the maturity curve, the agent maturity curve, because what we found was people were still, even after all that push, people were using it to do very low level tasks, not really jumping the chasm and making the agents work for them. In a matter of a month or two, we were able to push a gigantic amount of our engineers from just doing basic things to like P3 and P4.

4:01 Which is really using agents and an agentic way of doing coding. I think that was pretty dramatic across all our stacks. We could clearly see the output go up to 2x. And with quality being higher, it's not like 2x more bugs or 2x more issues. What are the steps in this hierarchy, how does this work? How does an engineer move from the first rung to the second rung? You have just simple utterances. That's where everyone just starts, simple utterances in Claude Code go do this.

4:31 But they don't know what good looks like. They're not expressing what good looks like. And then, it kind of handles some of the simpler tasks. But as they get into more long horizon stuff, they have a longer horizon question, they'll find that, it doesn't deliver. because they didn't express what good or bad look like in the utterance. They didn't build a comprehensive plan, like doing all these behaviors. And so these steps are kind of, what should you think about?

4:55 And then it moves into teams and sub agents and how you should let the harness help you orchestrate decomposition of parallel work and, manage work trees and really, crank up the parallelism and then it kind of ends with autonomy and automation. Where you're driving all of these agents in these work, managing these workflows across SDLC tasks. This is test failure happens in CI/CD, then an agent will pick that up and fix the test failure, for whatever reason that was.

5:24 And then, another one will pick up the code review and merge that change. So like, you know, all these different types of tasks and automation. We started with, I've started with everyone in the very beginning. But now we have thousands of people in this kind of autonomous mode, I think our bigger challenge was also enablement. So explaining that concept of how getting to P3’s or P4, it's not a zero sum game. It's truly like you're moving into a new realm of how you use agents.

5:50 I think that was a big enablement thing as well and learning for everybody. How did you approach that? Inspiration and real data. Yeah. Yeah. Lots of people demoing how they, this is how I work. I'm going to solve a problem and I'll solve it live. And just being very, very open and having a bunch of different people show their version of that. And also every cloud ended up having some sort of an innovation that they created using sort of agents that would have taken them months, which they could do in a few weeks.

6:17 And these are real shipping products that you would sort of be announcing at Dreamforce. I think that also helped a lot for people to see that, oh, actually, this thing works. It actually is helpful and it can make a difference. One of the architects in our team, she said, normally it would take me several days if I have to understand some new code, because I have to talk to 3 or 4 different teams. So just the fact that it eliminated the meetings to understand someone else's code was just huge for them.

6:45 And then the second thing that was more organic also, that really made people spend an enormous time in incidents or sort of existing management and having these agents really give them the right information I think that was another huge groundswell. Every team has now their own incident resolving agents. I saw, going through a Slack thread where there was an incident, the outer service got paged, wherever the requests landed. But it was actually one of the inner services that had the issue.

7:12 The agent just said, here's the root cause, and here's the team and I have raised a page for them. We'll just transfer it over. And then this is the POC just Slack them. Imagine nine teams would have been woken up, by the way. It said, nope go to the other one. That is magical, right? As an engineer you're like wow. So are you just telling everyone, hey, come up with cool ideas, post demos in whatever channel?

7:36 The expert suite was just an idea of how do we take all the skills that they're producing and share them? That was the original idea. What's a marketplace? Can we do evals for these to show their effectiveness? Can we do an online eval to replay the sessions with or without, maybe having Claude suggest improvements to the skills to make it more token efficient or arrive to the conclusion faster. Whatever the change might be. And it kind of just built into this, uber skill marketplace where we can show that the skills that are being shared actually have real value on outcomes.

8:12 It’s part of our engineering harness in a way we can, kind of visualize, not just the coding tasks that we're doing, but you know, what is the different parts of the STLC that we're automating? How do we design workflows? And it allowed us to do the hierarchy thing because there are skills that are the shared sort of, overall team, overall engineering level, there are things at the team level, the smaller scrum team level, all of that stuff as well.

8:37 We have thousands of repos. And if you want to democratize the, you know, fixing a bug anywhere in the codebase, you have to know, okay, I'm going to go pick up these repos and this the API surface area I want to target. So like, how do I, what workflow do I use to actually address that code base I've never worked in before and then produce a quality PR. So you're letting the innovation happen bottoms up. Then as you see it start to work you help it scale.

9:07 Yes. What are the guardrails that you need to add? Yeah, maybe the most important to mention that we have to look at is quality. Taking a more scientific approach to, evaluating the capability and its usefulness in the different contexts. Just having a conversation as people. And try to understand what they're trying to achieve and what the value proposition, they think, they have? And as they sort of share it out, make sure that that's actually being achieved by other developers.

9:34 This becomes really acute because just in the last several months since we've sort of pushed for going faster, better. We've almost doubled our code volume. The risk we always said was, okay, does that mean there'll be 2x more bugs? 2x more cases? Thankfully, we found out that wasn't tracking. Meaning that was much, much lower. So we are, on the average, actually less sort of bugs per code that's being generated while our code volume has spiked.

10:04 And that was because of all the guardrails, Ricky is saying, making sure there is checks and balances, in terms of what people do, making sure that all the traditional sort of mechanisms, as sort of new mechanisms, holds true in terms of the quality of the code. For anyone listening to this that wants to repeat this exercise, how do they do it? Are you just going through commits and labeling, was this a bug? Was this a bug?

10:27 How do you make this quantitative in this way? Everything's already all labeled, and everyone uses the same sort of, standard SDLC practice. So they do use the ticket system. Everything is labeled consistently. So there's a fairly well curated data set that we can start with. And then, that's all linked out to the PR’s. That's another big thing that Richard and team have created is really a benchmark eval set. Now let's say somebody wants to try new skills or maybe wants to try a new model.

10:57 Like, we want to go try it out and say, hey what's the ROI for the model? So we are able to now simulate with all of these how would that model work? What should the cost be? Is it better? Is it worse? And that's been very, very helpful for us. And what frontier did that model change unlock. What can we now get out of this model? All the short and long horizon tasks across the PDLC that we have created benchmarks for, we just replay them.

11:19 Plus the new thing we've added is the fail to fail. So like, what tasks have models consistently failed at? The long horizon ones. So that when we see them passing, we're like, hey, that's the reason we should pick up this model, there's a bunch of things we can now do to capture those capabilities. Richard, you had a quote that, autonomy without validation is confidence theater. What proof of work do you make an agent show before you believe in it?

11:50 And is it the same for coding and non-coding? I try to find some objective criteria by which we can measure the task that I've given it. This is what bad looks like. This is what good looks like. And so bad would be a negative test case. So like, did you break anything else? Did you add fields to an existing entity when I asked you to go create a new schema? Did you touch anything in an existing schema?

12:13 Just looking for just, objectively gradable criteria, that I would normally look for. So like, what features exist in the output that are indicative of quality and or indicative of a failure? Then instead of just asking an agent to go and review itself, it's like, did you go find the features? Show me the evidence of the features, show me the evidence of the negative features. So just show the sources. Yeah. Show your work. Build a matrix of all the features that you found, and then we'll go into the details of it.

12:43 If not, go back. How do you decide what size task to give an agent? You have a bunch of tickets, let’s say you're doing story points, there's 1 point, 2 points, whatever. As the model gets more intelligent, how do you decide what's the right altitude? What's the right size task? More story points is usually better, because I feel like the amount of work that you have to put into a very small task might not be worth it.

13:06 So it would be cutting a much larger set of features altogether. And then having the model or the harness, you know, figure out how to decompose, working with the model, decompose that into smaller, isolated tasks and for execution. But anything that's going to take a few minutes, in my mind, is a quick interactive session. And everything else, I just try to accumulate into a big chunk of work. As humans, we've been trying to decompose all these tasks.

13:33 Like go build this big feature and really coming down to, Oh, do one, two, three story point units and I'll go execute that. But if you have a team of agents working for you, it can do the decomposition. We don't need to do all that decomposition for them. The better the plan upfront the more output you're going to get, the better output you're going to get. What was your biggest mistake rolling out Claude? Over the last few months, over the last year, what's the one thing that you got wrong?

13:59 Use the right model for the right task. That gives you the right ROI for the right price. The length and the complexity of the task. Right. Because we saw people just using it to go generate regular files that they don't need a heavy duty model for that. So length and the complexity of the task is how we look at it. What are some examples of a Fable problem versus a Sonnet problem? There was a bug that we were aware of in one of our products.

14:28 We could not find it. We could not find the root cause. I think there were a couple attempts at an investigation just to find and fix the issue. And we deployed Opus on this and we had it debug the code, make changes, compile it and debug and try to find it. And then after 24 hours, it found it and it came up with a proposed fix. Now I think the validation part of this is the most important part, because it's a pretty critical piece of business software for many of our customers.

14:55 And so getting a wrong answer here would impact financials or whatever else. So it was a small bug, we just don't want to have the fix turn into a big bug. And one equally as hard to find and fix. I think that's a great example of where this, of a long horizon task that is suitable for a very large model. Richard, what's your one piece of advice to engineers that might be listening? And MK what's your one piece of advice to CTOs?

15:20 Be bold would be my advice. And it's important for us to you need to think about moving from a system of engagement to a system of agency. Because you got to let innovation happen. It can't just be top down. It has to be both top down and bottom up with the right guardrails. And I think that is a big thing and create that learning environment where people can share. I think those will be the big things I would advice.

15:44 I found that I've always created a backlog of things I want to get to, and I can burn down that backlog very, very rapidly. Even if I need to go meet with the team, what have they done? What changes have been made in the code base? I can just do so many things that I never thought were really possible. Dig in deep. There's just a lot of discovery here and there's a lot of learning, and there's a lot of fun to be had in this process.

16:09 Thank you guys so much. Thank you. Thanks for having us.

© transcribe · For agents Built with care and craft by Gokul Rajaram