transcribe

Building the Claude API for the Frontier

Claude · 18m · transcribed 5d ago
More from Claude Business
𝕏 Share ▶ YouTube 📥 PDF 🤖 .md

Transcript

0:00 A lot of roads lead to chrome caching, not all, but for any customer that we talk to that caching is the first recommendation we get. so we're going to be talking about the messages API today and and just cloud platform more generally. for folks who have not built on top of cloud platform before, what is the messages API? >> The easiest way to see it is the cornerstone of our API. So it allows you to customize it and build almost whatever you can imagine with it. So we have this very normal structure that anyone who has been building in the industry is used to by now in which you have a system turns and user turns and tool calls in the middle. On a barebones perspective, you can take it from there and build whatever you want. But into the API itself, we have built a lot of tools to allow you to further customize and have knobs into behavior. One of those can change behavior pretty dramatically. Like the effort you take the same model on high effort and low effort feels like choose hybrid models.

0:54 You can from first principles build anything you want. But with all the tools we have put into the cloud API, we're giving you a lot of like an easier path to get into the ultimate app you want to build. Some of it comes from us having a very opinionated take. So we're very close. We're in the same room of research, right? We share building, we share a lot of Slack conversations, a lot of meetings and with it we have a lot of a good take on what actually makes cloud shine. We put those as the default in API. So if you just take and you to copy paste from our documentation the most basic request and it's going to carry what we think is the best out of the box experience but with it there's also like all those knobs so such as effort that you can try to prompt cloud into having a certain type of behavior or not but with those knobs you can do more than any prompting outside could ever do. So it actually unlocks a level customization that is just not available otherwise. Talk to me about how we decide sort of like what to ship like what knobs and dials do we allow to actually unlock the the best cloud intelligence. It's a it's a very two-way conversation with the researchers at this company. So one way comes from them to us in which they are experimenting and building and they start finding out things that cloud is good at and that comes as a very raw thought like oh you know once upon a time a long time ago they started noticing that cloud is getting better at using computers. Yeah, I remember that day.

2:06 >> And then ultimately that became the computer use tools we have in the API. But sometimes it goes the other way in which we are talking to our customers and we're understanding what our customers want and ask us more for and we convert those into actually asks for research to train into obviously not the past clouds but the new clouds because there's always a pipeline. So any user ask or user need we we have we can eventually put in the API and that conversation keeps going. It's an ever you know it's a good good chunk of my job and it's how we actually try to get that best cloud out each time.

2:33 >> Yeah. and you know what what have we done to make a choice that we felt like really you know empowers developers to to bring the best of cloud to their end users as well. >> It's funny because the these two feel so small but they are very big in practice. So being able to change effort in system prompts mid conversation are two things for me that really show that power of being able to have a knob later on and you could always change a system probably you could always change effort but then you break prompt caching and that to a user is like a dealbreaker.

3:00 Yeah, because that's the unique thing that we can provide there which is okay, how do you make these changes? How do you you know empower the the experience you want to while you know keeping cost low or otherwise you know unlocking the best intelligence. >> Exactly. And you know sometimes a user has an agentic flow that has being operating under high effort and they know that the next request is actually easier but if they change the effort to low the cash is broken and that defeats the whole purpose of their saving cost.

3:24 But being able to change effort mid conversation really gives you that power. So we're we're making these changes. We have like this tight loop with research, with inference, with a bunch of other teams, safeguards included. What does it take to get from idea to launch in the API and empowering you know our our users to leverage cloud? >> After the idea, the first thing you need is some sense of how well cloud works and that usually goes around evos and each step of the process. This is what I was hoping to get into like you know how do you like we have all these changes coming through like what do we need to do to ensure that one workflows aren't breaking or otherwise you know you're getting the best from from any of these knobs and dials we're bringing >> yeah and we run a lot of evos and we recommend each customer who is adopting each of these knobs to run their own evals because you know ultimately we are an API we're very generic we're trying to serve everyone and for a customer building on top they have their own use case they have their own personalization that applies to them and only them in the world >> so we we talk a lot about changing things in the API as a result of unlocking intelligence sort of staying at the frontier with cloud but you know traditionally APIs have been versioned there are breaking changes and you know we think about these things in a way of migrations it's like okay user X needs to make a change in API Y and so on and and roll that out to their users it takes some time with models has actually been quite different so describe like you know in in your mind as you're thinking about new changes in the API how do you actually ensure that you know migrations are relatively slow you know that that rollouts are relatively fast and otherwise you know users can ingest these changes well.

4:53 >> Yeah, I'm very curious about your thoughts on this as well actually. >> yeah, I I think one thing that's been very interesting is you know I I come from other worlds in other roles where API versioning has been relatively important but also relatively painful. models provide such a an interesting version mechanism for us, right? I mean you have this model string, you have an API version, you have an intertwining of of these two things, you have prompt stability, you have a buckets of of changes or switches rather that that allow you to bring new changes to to your end users. And it's actually been very fortunate that models have been able to be really on this frontier of intelligence. And because of that, you know, users are more willing to to make adjustments because the value is so high for making those adjustments.

5:37 And I think that is such a powerful mechanism both for our users but also for us to ensure that we're bringing the best intelligence forward. And so, you know, we we think a lot about, you know, how to make sure our our API versioning strategy keeps up with with model intelligence, but also how model strings effectively as a version mean that we're bringing the the best intelligence very very quickly to users. But again, it comes back to what you just described, which is your harness still needs to be eval. It needs to be tested.

6:02 It needs to be validated that that these changes are are are working well. you know anything from number of tokens used to intelligence to outcome from the task are all measured and and I think are are very very important for any end user to to really think about. Yeah to to elaborate on something you said there sometimes we need to do you know some implicit versioning when you go from one model to the other and the easy part of implicit versioning is when you're adding new capabilities no one complains it's difficult sometimes when you need to deprecate one capability someone relies a lot >> you go in deprecation I did additive right >> yeah one that comes to mind is when we changed the thinking to from regular extended thinking that everyone was used to to adaptive thinking we knew by then that adaptive thinking was already performing better in evos and we wanted to roll it out and we wanted to be you know the way to do thinking for in the future. So, we did roll it out, but for a while, we kept it as this other type of thinking that you can still switch back as a developer, toggle back to the old thinking.

6:55 >> And I didn't even touch on these. We have beta headers and and other mechanisms for you to roll back to previous changes if you need time to update your harness. >> Exactly. And we tried to to buy that time some of it at some point we need to do a harder application because at some point model stop being trained for this old way and then for us to put a nomin API that doesn't do anything or doesn't do what it's supposed to would be even more harmful to the developer.

7:17 So that's the end date of the migrations essentially but we do try to smooth it out as much as we possibly can. You always have a model or two to slowly migrate and we change what is recommended but you can still go you know stay your old way. >> Yeah. >> so maybe stated plainly, you know, how do you think about the sort of loop of capability comes out? Like what should a developer think about as they're up upgrading to these new versions? Like what does that sort of loop with their emails look like?

7:40 >> Yeah, I think in general, you should like apply this thinking for every single part of your workflow. Even as you're using an app, if you're using cloud code, it's always good to periodically reevaluate all the loop and setup you have around your own product when models change because models can be really good at one thing that makes one part of your harness different. And this is actually goes into a whole conversation about building harnesses in general which they're very good at leveraging this capability from the model. But at some point if a model is 10 times smarter than the previous model was then the harness can start to be a limitation. So I think even this force deprecations at times are a good forcing function to reevaluate parts of the harness that maybe don't need to be there anymore. Whether this new feature can help you get better prom caching and if you get better prom caching you get better cost savings. So you know we we do a lot of things on our end lot make a lot of changes make a lot of updates for intelligence we also make a lot of them for cost as well and you know prompt caching is one of the the largest levers that we have in in order to to control for cost. What are some other you know ways that you think about like cost controls in the API and otherwise prompt caching is is one of our unique value propositions that that we can control and help our users with. I think a lot of roads lead to prom caching, not all, but to take a second to talk about prom caching itself. for any customer that we talked to that still hasn't optimized pro caching is the first recommendation we give because the one that can bring your cost down meaningfully.

8:58 >> And and what is prompt caching? Like how does that work? >> Prompt caching is a good way to see if you're if you're in the middle of an injective turn in which your second request is very similar to your first request where it's more like an append. You're just adding new things each time. We cache the blocks that already run once whenever you know our sales teams are talking to customers. the first thing comes up like have you how much have you worked on your prom caching rates?

9:18 >> Can you get it to up to 90% prom caching cuz that those are like dream numbers to achieve and it really translates to cost and it's a win-win all around. The other ones are like and you can tell me what you think about this in general like how do you approach cost per task versus cost per token because some of those features we have they don't change the cost per token of the model but they change the number of tokens ultimately used. It's a it's a tough one because I I think you know tokens and in and cost per token is just the unit of measurement that a lot of people look at but a lot of times we found in many evals and so have our our own users have found through many evals that the cost per token also needs the dimension of intelligence per token and intelligence per token is how quickly can the outcome be achieved as a result of the model that you've chosen. So, you know, one of our higher intelligence models, maybe a fable version versus a haiku version is going to be wildly different in terms of the number of tokens to use to to achieve that same outcome. And so, I think we're almost back to this eval point of, okay, great. How do you make sure that you're harnessed, the model that you've chosen, the knobs and dials that you've pushed on, pulled on, however you want to frame it, are tuned and tweaked well enough where you have confidence in your eval story that oh actually if I upgrade to this new model or change this dial I'm thinking or or otherwise that you're still achieving the same outcome but potentially on a different cost curve one that's favorable to you. we see that a lot with you know users using you know Haiku or Sonnet in some cases which is great but then not realizing oh actually if I were to just use one of these higher intelligent models that that it'd be in in a much better place.

10:50 That's something that I'd love to see more of and and you know even have us build product around and and help our end users do do that even easier than than they can today. >> The biggest is when the the stronger model is more expensive but then it solves a task that you couldn't solve before. >> Maybe the inverse is true too, right? where as the models on all classes get more and more intelligent, you know, there are times where you might use a model with slightly less intelligence that is now capable of of handling that same workflow.

11:15 >> Yeah, exactly. So, maybe you're using Fable 5 or something because it's the only model that could solve this, but now Opus 5 can solve it for you. >> Yeah, exactly. Well, let's let's maybe shift gears slightly. I would love to talk about, you know, we talk about the API, we talk about leveraging for intelligence, we talk about eval flows. What we haven't really talked about is how to think about good actors versus bad actors potentially in in misuse of the API because I know you know a very top of- mind topic for most folks that are are using these APIs in in safety as is our number one mission you know as anthropic. So how how do you think about the the difference between good acting and bad acting in the models and maybe some of the changes we've been making recently?

11:55 What are those intended to protect and why is that so important? Whenever we get a new model, we start seeing how smart it is. We get the the first feelings that this excitement like, wow, this model can do things that it couldn't do before. And the thing that I always tell myself and everyone in this company tells themselves is we have a lot of ideas, but we have such a large pool of customers that they're going to have more ideas than we do. So, we take this model and it is our number one mission to also like get it in front of users so they can build all these amazing products. We have users who are doing science research with the model.

12:21 So, they need to get their access to this. And a second instinct is the the other side of that in which you worry about like what will a bad actor do with this model? And you know navigating that balance is the focus of a lot of people is one of my focuses and you see it in the very clear split of the myths and fable models. >> Yeah, let's talk about let's talk about that. >> You know, every model in the API runs with a lot of classifiers and a lot of those classifiers is trying to prevent bad usage of it.

12:45 >> Yeah. >> You know, it was born on the fact that there's a lot of cyber good to be done >> which we believe. I mean we like our our mission is safety to bring best intelligence, you know, best outcomes to society based on that intelligence. how to cure the next disease, how to, you know, ensure that we're having a better blockage on on cyber security issues. You know, there's there's a bunch of good to come from it.

13:05 >> Yeah. Exactly. And it is very important for a mission to actually get the right people with access to meth. And then Fable is the other side of this coin for anyone who doesn't go through this entire like KYC process can actually get and use Fable. And it's the same intelligence, the same model weights at the end of the day. And it's just about how much you can do with this model based on classifiers. The other part of the safety coin is there's a lot of distillation that happens. If we put a model out there that gets distilled by and created into another model that sure will be weaker, but it'll still be a strong model.

13:36 That model being unsafeguarded in the wild is an incredibly dangerous position for the world and it is something that we work a lot on to try to stop. And we're very forthcoming and honest with users about this. So we just rolled out a batch of changes on thinking protections and we documented it as measures we need to do to prevent the isolation. >> Yeah. How do you make sure you know we we are balancing the developer experience with some of these safeguards? What are some other what are some mechanisms that we use to make sure that that happens?

14:03 >> So we run an absurd amount of evals. Everything that eventually makes it public. There were 10 other variations that we experimented internally that we're just trying to see what is the actual best one and it is a difficult problem but an extremely datadriven solution and it takes a lot of people looking at a lot of numbers over a long period of time to eventually make one decision and that's one feature that makes >> it's actually quite fun to to look at all these numbers as well right and even getting to the point where the model is maybe so good at one of these evals even in the face of some of these changes that we're making that we actually have to make the evals harder in order for for the model to you know, continue to hill climb or or otherwise improve.

14:40 >> Yeah, eval saturation is a very funny good problem to have that we keep running into. We had an eval at some point it stops being meaningful cuz every model can solve it. >> Yeah. You know, maybe what's what's one thing that you're excited about for the future when it comes to model launches generally that you can share of course things that you know maybe more broadly you you'd love to see and continue to push on.

15:00 >> The most exciting one is always new capabilities. So there's always this drive internally whenever a new model is fresh out of the oven. It's like how what can you do like can you test all the prototypes that you built before that we're failing and I think ultimately it's all about model intelligence. Everything else next to that matters but nothing matters as much as model intelligence. So what we do a lot we reevaluate all of our features.

15:23 We reevaluate many features that we didn't launch before cuz they weren't good enough. We always have a few features in our set of options that we haven't launched yet because it's just not at the performance level we want to put out for users. Yeah. >> Every time we put something in API, it needs to be, you know, has a stamp of quality. We really care about it. >> Yes. >> And the thing that excites me the most when we have a new model coming up is rerunning evals and all of those like do any of those clear the bar now? And when it does, we actually put it out there.

15:47 And it's something that we also try to pass that same insight to our users. Like all these things that were very close but not there before or they're there now. >> Yeah. How are you thinking a lot about long running and evolving the API to power these autonomous flows as we see AI you know running into all these workflows a lot of our API is designed at the end of the day about for agents so you know if you open our docs you very quickly see about our cloud manage agents offering and there's a lot that can be said about that but it's it gives a really easy way to just spin up and have very very powerful agents but then the other side being as we talked a lot about the cornerstone the messages API itself we have a lot of features there but most of them they are at the end of the day about agentic flows. This comes from us talking a lot to customers who build agents.

16:33 >> Yeah. >> The other tools we have are a lot of in our client side tools and they are things that we know work well and this comes from our like our huge battery of evals and you can always use your own bash tool or text editor tool but you can also use ours and the others being all those like toggles to control intelligence. Obviously you can control thinking you can also control effort. >> Mhm. And those matter because they really change your agent from like being able to do something or not.

16:56 >> Yeah. And I think a good iteration of this too or or a good showing of of what the messages API can do is through all of our first party apps. you know, all cloud code, cloud AI, but also we we have cloud manage agents as well. You know, it's been very great to see a lot of this, you know, happening where we can take off a lot of this from your plate as a as an end developer using the cloud platform making that easier and easier for you as you go as you go into these higher order agent abstractions, which is really cool. And so yeah, prompt caching, there's, you know, thinking tuning, there's consistently looking at changes we're making to improve efficiency in the API and and and otherwise making things faster.

17:33 so once a lot of those things land in and are launched, I think we're going to continue to see this evolution more towards agents and and leveraging the messages API to to make that happen. >> Yeah. And actually cloud manage agents is built on top of the messages API. So it's also another case of us just being very tightly integrated in house because we are building this agents API we at the messages level understand a lot of what they need and when we understand what they need we put it in the API available and everyone being agents whether through the cloud manager agents API or just straight from messages API gets all the same lessons to them as well.

18:05 >> Yeah makes sense. Well it's been awesome talking with you about messages about the API and about just cloud platform generally. We have a a lot of work ahead of us. but you know, it's very exciting to to hear about the ways that you think about model launches and and evaling and just making sure that everyone can get the best intelligence from from Claude. So, thanks. Yeah, great chatting.

© transcribe · For agents Built with care and craft by Gokul Rajaram