Why Monetizing Agents Needs Different Infrastructure - John Yeo and Charlie Lamb, Autumn

0:00 My name is John. I'm the CTO of Autumn and this here is Charlie. He's our founding engineer. And today we're going to be talking about why monetizing agents need different infrastructure. And so Autumn is you know, as we said, billing for AI companies, but what does that really mean? I think the the way I like to think about it is that we're billing for modern companies. And so historically, you know, like a decade ago, billing was very much just subscription-based. You know, you would just charge something every month.

0:31 and then, you know, a couple years back we got into this interesting period where it was very much usage-based. So, all of these infra companies started popping up and you would have you would basically want to charge based on say compute hours that you you would accrue over the entire cycle and charge through an invoice at the end of the month. But we're sort of seeing an interesting pattern come with AI applications where billing is now a lot more tied with your application. You know, you have these things like tokens and credits and actions that you want to check for up front before a user actually makes a request. and that requires sort of a different infrastructure, right?

1:14 And so that's what we're going to talk about today. We're going to give you a sort of a a preview of what that looks like under the hood in hopes that, you know, maybe you could implement this at your company or, you know, maybe just share some tips in terms of how you can solve some of the problems that we've seen other companies face. yeah, so to set the scene, you know, imagine you're sending a message in CodeX and you see this error. You've hit your usage limit. You know, I'm sure all of you all have probably hit that if you've been token maxing enough. But on the surface it seems very simple, right? You know, maybe it's just tracking Redis as a rate limiter or it tracks Postgres.

1:51 But today we're here to show you that there's actually a lot that goes under the hood, ex- especially when you think about high-throughput concurrent atomicity. And yeah, so now I'm going to let Charlie go into the first portion. >> Cool. So, we're going to start with a very simple example. We're going to be building out a billing system for an AI coding assistant. We're going to give all of our users 100 messages per month. We've already got a customer's table in our database. So, to facilitate this, we're just going to add a single column and give all of our customers a balance of 100.

2:21 Whenever we go to call the model, we're just going to do a simple process transaction, check the balance, and then track the usage from it. Now, this gives us safe concurrency without needing to call Redis or some sort of external queue. This is a very basic billing setup, but for this use case it works pretty well. And we're going to see how it struggles to scale. So, let's say for example, I want to add promotional codes to my application. And if promotional codes give me a lifetime balance, this is different from the balance in the database right now, which resets every month.

2:52 So, how am I going to do this? Well, I choose to add a new column in my database. I'm going to call this lifetime lifetime balance, and this is going to reset separately from the original balance. Now, to be able to facilitate this, I'm going to have to add more application logic in my code base. I'm going to have to do a database migration. And overall, the change is quite complex. So, you can imagine if I then add like auto top-ups, if I then add rollovers, this is going to get like a very complex and have to do lots of migrations and lots of changes into my application logic. So, we should take a step back and think about a better way to do this. And that would be moving to a row-based approach. So, let's create a new table with balances. And we're going to add a row for every single balance.

3:30 So, for our monthly balance, we're going to add a row. For the promotional balance, we're going to add a row. And for any auto top-ups, we're also going to add a row. And all these balances have their own expiries and their own metadata. So, this does make tracking a little bit more difficult. We now have to track over multiple rows. We have to choose which balance we want to deduct from first. We can do this with a consistent algorithm.

3:48 Say, for example, I'm going to take from the most recently expired balance first. And yeah, this is a much better system. So, now we know exactly what our balances are. We now have a different problem. We don't know how they got in this state. And to fix this, we're going to add a ledger. So, this ledger is going to give us all the historical information we need to know about our balances. Every time we delete a balance, every time we create a balance, and every time we track usage from this balance, we're going to add an event in our ledger.

4:14 This means at any point in time, we can go back and see exactly what the balance was and how it got there. Now, for us at Autumn, this is really useful. We have a Slack bot that all of our customers use, and this will like go through the ledger, see exactly how the balance got there, who added the balance, who deleted it, and it really relieves us of a lot of customer support work because we have a ledger which says exactly how the balance has gone in that state. Now, when you have a ledger like this, it can be quite tempting to calculate the balances from this ledger.

4:38 So, I can create like a materialized view or just aggregate over all the events in my ledger and I calculate the balance like this. This is a good approach if you have like some billing which you calculate the balance at the end of the month, maybe you're showing something on a dashboard. But, for our AI coding assistant or for how we do billing at Autumn, it doesn't quite work. So, we also need a counter, potentially in Redis, which has the current value of the balance.

5:00 And we can check this value very quickly without having to do any complex aggregations or anything like that. So, using the two in tandem is really the best architecture in this case, and that's what we'd recommend going for. >> All right. Yeah, so you know, Charlie mentioned how he sort of laid out the groundwork for like our credit-based system. You know, you have Redis as a counter, you have a ledger which, you know, it might be stored in ClickHouse.

5:21 But, now I want to sort of focus on the case where you have very high throughput and each of these requests need to be handled atomically, which is probably the case for a lot of you guys if you're building, say, an API, right? Which is, I think, very relevant in today's world of AI. You know, everyone's opening up MCPs, and you know, everyone's handling tons of requests per second. And so, conventionally, the way you would solve high-throughput scenarios is that you would simply take each event and send it to something like a queue, and this these events would then be processed asynchronously, right?

5:54 but the problem with this is that with AI, it warrants sort of a different framework because with a asynchronous method like a queue, a lot of these event deductions happen at in a background process, and so that means there's a lag. And you can And you can imagine in the case, if you want to go to the previous slide, if you have several agents that are just like spamming concurrent requests, each of these are not going to have the real-time balance, and therefore the customer is going to be able to go into overdraft, and a lot more credits will be used. And so the architecture that we see a lot of people adopting today is something more like what we call a lock and release.

6:36 And so you can imagine something like with ChatGPT, when you send a message, it will first estimate the number of credits that your message is going to use, probably by the length of the message, the estimates the number of input it will then deduct that atomically from your balance. It will process the request, and then following that, it will either refund if you've used less credits than what was predicted or deduct even more. And this way, you know, if you have five concurrent requests coming in, let's say each one costs 100, and you only have 100 credits, one of those requests will go through, and the other four will be rejected. And so this way, everyone's balance is very much more accurately handled. One thing I want to talk about here is that infrastructure-wise, it makes sense to do this with a Postgres transaction. But we're going to see later on that billing systems get very complex, and I think if you've worked with one before, you probably know that there are many joins across many tables, and it's not as simple as just saying, "Oh, let's lock one table, perform a transaction, and then release it afterwards." And so the way we've seen work generally quite well for many companies is that you can use a Redis cache and it's atomically deduct credits via Lua scripts and this is much faster in terms of reads and writes.

7:50 And so sort of to wrap things up, we've talked a lot about credits and what I want to talk about now is how to make your billing system flexible which is very relevant today as well because AI is very unknown and you don't and a lot of people don't really know how to price it and I think you've probably seen by now that people are making pricing changes at like 3x or 4x the rate that they used to.

8:13 And if you've ever done one of these yourself, you know that it's actually like insanely difficult. And so hopefully we're going to give you a bit of gold nuggets here on how you can make that simpler. And so the first frame the first mental model that I would try to hold when thinking about how to create a flexible system is that billing is relational. Now a lot of y'all probably think that with billing all you need to store is your customer state. You might have one table with each customer user ID, credit balance, plan and whatnot, but that is only treating one part of the problem. Ultimately with billing every part of the system, the plans, the entitlements that your users get and the customers themselves all change. And by designing it as a system like this, it becomes much more flexible and much more I guess seamless to create billing pricing changes. So you can imagine if we wanted to create a new version of Pro, all we need to do is add a new column. All of our current customers are grandfathered and whenever we want to switch them over, we can simply change the foreign key on the customers table and likewise with entitlements.

9:17 It's much more cohesive. >> And yeah, so I think it's also important to remember that your billing config should be hierarchical. And what I mean by this is if I want to create like a usage limit for example, I want to limit all of my customers to 10 messages per month. Now if I wanted to do this and add it to all of my customers rows in my database, this would be a big change. I'll be adding thousands if not millions of changes to my database.

9:40 The best way to do this is to have some sort of global global configuration in which all customers use. And then let's say for example, when I add more complexity, I want to introduce a pro plan with a different usage limit. Let's say 25 messages per month. It makes the most sense instead of changing all the customers to add a configuration at the plan level. So, I'm going to have this config at my plan level and I'm going to have some sort of logic in my application which goes through it first, looks at the customer, looks at the plan, and then looks globally and says, "Is there a usage limit?" And this enables us to resolve it in a much more like a hierarchical way.

10:11 And let's say for example, the CTO wants unlimited usage. I can just add a configuration at the customer level, give them unlimited usage, and a hierarchical structure works really well. So, yeah, thanks for listening to the talk. We've covered a very simple billing system. We just added a single column to the customers table. And we've added complexity. We've added more complexity to our product and also more like load and scale to our system as a whole. And yeah, hopefully you learned a lot and thanks a lot for listening.

10:40 Thank you, guys. >>