How Revolut Cut AI Coding Costs 90% With Self-Hosted Open Models - David Abitbol, Revolut

0:12 >> My name is David. I lead AI operations at a company called Revolut. We have 50 minutes today together. We'll talk about high-performance engineering and self-hosting open-source model on dedicated GPU. How's that sound? Cool. So, it's going to be a technical one, so bear with me. so, maybe to start off because I know this is the US and you may not know Revolut. quick words about our company.

0:44 We're a digital bank in Europe. We have 80 million customers. We're live in 45 markets. With the most downloaded financial app in Europe for the last year. And as of July, we're Europe's largest private company. So, we're doing okay. The the emphasis of Revolut is, you know, shipping a lot of product and so there's a high focus on engineering culture and excellence on that domain.

1:16 And so, we'll talk about that. So, let's talk about some AI. We have a chart here. I going to see two lines. This is a sample of 2,000 engineers performance over the last year. So, the green line is going to be lead time. So, it's going to be when engineer picks up a feature until a feature is merged and ready to be deployed. The blue line is going to be coding time. So, the green line is in days. The blue line is in hours.

1:48 Just to reset the timeline in case people forget because AI moves really fast. Cloud code came out around February 2025, right? And that's the Cloud Code Sonnet 3.7, the one would delete your prod database, right? Not the safe Cloud and Codex that we have today. So, bear in mind when we talk about adoption. So, what you see here on this chart is January 25th, first introduced AI in the company, right? Gemini. First time people were able to use private data within AI.

2:21 Around February, we start playing around with Cloud Code. And then January is when we first launched Cloud Code at scale. What you see is Gemini had 10% improvement in lead time, 20% improvement in coding time, so that's people no longer needed to look Stack Overflow and read documentation. But then Cloud Code not so much impact at the time. So, when we launched it, we were highly skeptical about the cost of AI. We were seeing engineers spend a thousand a month, ten thousand a month with not so much ROI.

2:55 And so, when we launched it, we launched it with strict budget limits. And so, we're looking at the next thing, right? So, how can we provide unlimited tokens to engineers so they can use AI guilt-free for a fraction of the cost. And so, we looked into open source. So, around April 26th is when we started self-hosting open source model. We're a big fan of GLM for coding. So, around that time what you see, April 26th, GLM 51 we see 60% improvement in coding time, 70% improvement in lead time. So, dramatic impact.

3:33 What do we learn from this? Well, there's more impact on the lead time than on the coding time, so what we find is the bottleneck for engineering is not necessarily coding. You can save a lot of time not having to dig through your inbox or messing with Jira and Confluence. And so, when you have unlimited token, you're free to do that. You no longer have to save your tokens for end of the quarter or the most difficult task.

3:59 So, where do we stand? Well, open source is now a priority at Revolut and yet the economics are favorable. The billing is flat, we don't have any bill shock at the end of the month. And the French here is the exception. So, obviously, we keep using CodeX code code. The latest models are fantastic. Jimmy and I are going very well as very good as well. But, we keep them as exception. So, how do we get there?

4:28 Some metrics that matter when you're self-hosting model. So, to serve 2,000 engineers, we roughly need 56 B200. And that's to host GLM-53. GLM is a big model, right? 800 billion parameters. What you want to aim for is sub-3 second time to first token. This is empirical evidence. As soon as you exceed 5 seconds, my Slack inbox starts to blow up and so we know that's the limit.

4:59 In terms of throughput, it doesn't really matter as much, right? So, we serve in terms of interactivity, 200 tokens per second. People are fine, you don't need to go higher. the way people work has changed, we no longer look at the stream of thinking tokens, people run tasks in parallel and check in when the task is done. But, if you want to have high throughputs, when you're an engineer at Revolut, you just connect after 5:00 p.m. when no one is on GPU and you can get 800 TPS and then, you know, it's a dream.

5:28 In terms of cost, in terms of impact, self-hosting GLM, our blended cost of token is 30 cents per million. And so, our invoice is 1/10 of what we were spending on Anthropic and OpenAI. So, there's a lot of tuning that goes into running open source models. If you just get GPUs and upload weights on a GPU and you start doing some inference, you're going to get terrible performance. It's because your workload is going to be very dependent on the parameters that you set.

6:07 So, if you look all the way to the left of that chart, the blue line is going to be demand, so that's request per second. That's going to be what's going to be the bottleneck for your latency for your GPUs. And the red line is what we're optimizing for, that's the time to first token. So, when we started back in April, you could see request per second around three, which is high but not so high. And time to first token over 10 second, which is very slow.

6:35 All of the data that you see is the same number of GPU. We didn't increase GPU count, we're still at 56. And by the end, you see in August request per second, we're about over seven, and time to first token we're around 1.6. So, we've had 88% improvement in performance for 76% increased demand. >> >> So, the learning from this is there's a lot of parameters that you can improve on, and you really need to really scaling if you're looking to serve for some model.

7:05 So, let's talk parameters. There's no secret recipe, but the things to consider are the following. First one, long context is a scam. You really need to be mindful that even though you can serve 1 million token, 256 256 is really what you need. Quantization, if you're running Blackwell generation GPU, you don't want to run anything higher than FP4. And VFP4 is best. Roughly going from FP4 to FP8 is a 40% saving in GPU footprint for zero loss in precision.

7:45 Other parameters to consider, tensor parallelism, even though GLM fits on TP4, you don't have enough KV cache, and so maybe a TP8 works better depending on how large your workload is. And then the more esoteric, check your cache hit. Coding test needs to be above 90% cache hit. And you can look into optimize spec decoding, sequential pre-fill, PGD zag, all of these. Cool. So, now let's talk about production setup.

8:17 Gnome obviously is a text-only model, which is a limitation for some people. The trade-off I would say we would make every time is get the smarter model and figure out how to do image recognition another way. You can easily have a local router. Open Codex is one where you would route any image recognition to Gemini or Luna and keep Gemini for the main task. If you really need some vision model, Gnome 5G flash works really well. We have our UI designer making great use of it.

8:53 Turnaround for UI product has gone a really, really long way. And then the last one is going to be about the harness. So, all of these endpoints are going to be OpenAI compatible API with response API and so you can use them inside Open Code Codex Codex app as well. Works really well. And so, that's what we we recommend.

9:23 And then last message, this one is for the CIO in the room. There's been a lot of stigma in using Chinese open source model. So, the thing to consider is you're not using the inference dedicated endpoint that goes to China. You're just getting the weights and putting them on a US-based service. So, for data sharing side, this is totally fine. Revolut in particular, because we're a EU regulated bank, we fall under the most strict regime in terms of compliance.

9:59 And so, what we find that works is all of these coding tasks are not considered autonomous agents. They're always with a human in the loop. And so, how do you give reassurance that you have a human in the loop? You start by creating a sandbox that guarantees that. And the sandbox means now, if you've used Codex code, it has all of these features of approve for me. You just have to set the rules for the operations that you don't want to allow.

10:27 So, you can allow grab but not Gradle, for instance. And that's going to give confidence to your regulator that this is not an autonomous agent that needs different supervision. The sandbox also protects you from rogue use, right? You may have bad actor inside your company, and you want to shield for that. You'd be surprised what you find in a large company. And then, the last one is prompt injection. Not not so much a thing now given how smart models are, but always a thing.

10:58 In terms of data privacy, we spoke about it. So, as long as the data sits in one of the countries that are in the data equivalency framework, so US, EU, UK, Israel, you're fine. You can get your data to be used there. All of your inference provider or GPT provider are going to have zero data retention by default. And then, the last thing you need is a very strong open telemetry so that if you ever have an incident, you need to be able to go deep and look at the logs.

11:31 Also gives you some yeah, analytics. All right. Thanks, everyone. That was all for my talk. >>