How Sarvam AI runs its analytics infrastructure on ClickHouse Cloud

0:03 Hello everyone. So, who here does not know about Sarvam? Okay. Great. So, I'll let you know. >> >> So, we at Sarvam build AI for Indians and you know all of us who build AI at India, we are all Indians as well. So, we build AI for India. And this talk is about how we use ClickHouse at Sarvam.

0:34 How many of you are interested in knowing, apart from this, that what are the opportunities we have in the core AI space for amazing engineers? Show of hands. Great. We'll discuss this then, okay? >> >> Great. So, see these are the What Sarvam does is we have a GPU fleet, right? So, to train models, you essentially need GPUs and lots and lots of them. But so, we have India's first B200, which is Blackwell Nvidia Blackwell GPUs.

1:10 And Sarvam had the first Blackwell fleet fleet in GPU fleet in India. We have GPUs in multiple data centers. So, we manage this fleet of GPUs and then on top of this, we do training and inference. Now, the opportunities, we will cover the opportunities below the model layer, at the model layer and above the model layer, okay? So, below the model layer, the opportunities are when it comes to managing GPUs at scale, either for training or for inference.

1:44 I can tell you no one knows in India right now. So, that's the opportunity when it comes to managing the GPU fleet. Basically, GPUs are not as reliable as CPUs. So, what happens is they break. I mean, there would be downtimes with GPUs. And the other issue that we face is the entire stack that we have is also very new in terms of the training stack and the inference stack. What is difference What is the What's different over here is if you talk about regular back-end applications, then two containers would talk to each other over Ethernet. Right?

2:22 You would have all have seen that in Kubernetes, we use NIC, right? network access interface cards to do the communication because there the performance requirement is not massive. But when it comes to this GPU fleet, we either need Infiniband or we need EFA. AWS has a thing called networking called EFA. These are very fast networking because the way communication works across GPUs is when we are doing the inference or training is is through RDMA, remote direct memory access. Okay, so these are the challenges. So, if you are interested in systems programming, if database internals or operating system internals fascinates fascinates you, then inference or GPU infra management is something that you should go deeper into.

3:20 Even in inference, there are several internals. Like, for example, how to do prefill and decode segregated inference, how to do KV cache-aware routing, how to do the KV cache offloading. These are all separate separate I can say domains, and you guys can decide to become expert at even one thing, and even in that case, there are So, this entire field right now does not have any expert. And this this entire thing is so new that even the best engineer for inference would be just one year experience in terms of the relevant experience because the domain itself is new. So, I mean these are the opportunities.

4:01 So, what Sarvam does is on top of GPU stack, we have Now, we do inference, we do training, and then these are the models that we have trained. Now, so we have text-to-speech, speech-to-text, vision, and LLM model, which is mixture of expert this is India's biggest LLM model trained in India from the scratch. Now, when we train the models, these are the opportunities. One is the let's say if we talk about taxation in India, then for taxation, the completely unambiguous, clearly written pre-training data is required.

4:45 Then the basically how to file a tax in correct way, those traces are required. We call it SFT. And then RL environments where agent you know, we would give agent several different different cases use cases, and then agent would file the tax, and then judge would judge would evaluate it. So, this is called RL environment, okay? So, pre-training, SFT, and RL environment, this entire part is also core engineering, and there are lots of opportunities. For example, if I talk about taxation, then right now this tech for Indian taxation does not exist. This tech also requires us to have eval stack form for Indian taxes, which does not exist.

5:27 For US taxation systems, the eval does exist, okay? So, we don't have so, that means today no one can say that let's say Fable 5.1, how accurate is it when it comes to filing Indian income tax system, okay? Because those evals itself don't exist. So, these are some of the opportunities Again, below the model layer at the training level. Now, why are we building these models is because if we talk about India's 22 Indic languages, text-to-speech, speech-to-text, and OCR, and even when it LLMs, these are the best models when it comes to Indian Indic languages.

6:11 There are very, very few companies that support all the Indic 22 languages. From what I know, no company actually supports no model company actually supports all the 22 Indic Indic languages. But, that's the whole mission of Sarvam to ensure that we are building AI models for India so that AI becomes accessible to everyone in the country. Now, these are the products on top of the models that we build. One is Sanvad. So, Sanvad is AI telephony outbound and inbound telephony automation.

6:45 Next is Indus. Indus is the mobile app available on iOS and Android. You can install it. It is not just a chatbot, but also agent runtime. So, you can build your own agents. You can expose those agents on in your Slack. use it with your team in collaborative way. So, Indus is agent runtime. Then, we have API platform. So, all the models that we have, through API platform we serve these models. And Sarvam code is basically coding agent. And Cowork is again an application built on top of our agent runtime.

7:26 This entire stack that we have, we it's done in such a way, and that's where ClickHouse comes into the picture, that we only use open-source softwares so that we can deploy it in air-gapped environments. And we can deploy it in any cloud and for almost all the services we provide managed solutions as well. These are the five places where we use ClickHouse. First is agent observability. So the Indus or the agent runtime part that I just discussed, it is essentially Harness engineering.

8:07 When we do Harness engineering, the most important thing for us is to understand which tool calls are failing, which tool calls are taking more time, and then how many times is it failing after which tool call do they fail the most more often. We always interact with sandboxes quite a lot when we build Harnesses. So, what's the cold start time of the sandbox? What's the Basically, if the if the sandbox is already there, then what's the time of first sandbox interaction?

8:39 If a particular task that we are pushing to sandbox, and if it requires certain installations, then how much time does it take to install this dependencies? We use LangFuse to understand the Harness correctness. And it does provide the judgment part as well, so we can run agents in LangFuse itself to to evaluate the quality of each and every interaction.

9:11 Next is Signals. I'm sure you guys will be aware of it. So, our entire back end, front end, and infra observability is is on top of Signals, which internally uses ClickHouse. The next part is when you use Sanbad or API platform. The product analytics is actually the real product. By this, what I mean is when on someone our users come in and say that they want to make 3 crore calls, 30 million calls in the next 2 days time. So, one campaign can be massive campaign.

9:49 So, it's not possible for any user to understand the effectiveness of the campaign just by looking at the individual log traces. That's where analytics becomes very critical. Not just this. So, there are two challenges over here. One is it needs to be real time and it needs to be very fast. So, from what we know apart from ClickHouse, there is nothing else in the open source world that can that supports real-time analytics and sub-second latency analysis. So, that's why we use ClickHouse over here. And similarly, when you use our API platform, the the analytics where how many calls were 200 and how many calls you made, how many calls resulted in errors, how many calls resulted in success, what was the average time. So, all the analytics even for API platform comes from ClickHouse our ClickHouse integration.

10:50 We use for metering and billing, we use another open source system called FlexPrice. FlexPrice internally uses ClickHouse. And so, when you use any of our products, you there would be a usage screen and the usage screen is internally it's entire thing is even the bills are powered through FlexPrice. And the the usage screens and all the data is coming from FlexPrice. Even over here, the real-time analytics is very important and the sub-second latency is very important.

11:27 The last use case is data warehouse. So, when it comes to data warehouse, we have two specific use cases. One is we have massive amount of clickstream data, immutable events data, which we just push directly to S3. And then the on top of this on we have Glue Iceberg on on top of all this data so that because this is mostly cold data, okay?

11:59 What we have done is we create materialized views on top of this cold data in ClickHouse so that the query performance is much better. And for our rest of the backend databases, usually when it comes to creating data warehouse, the typical stack is we set up Debezium, we we create Kafka, and then from Kafka usually we create we write some Spark jobs or Flink jobs which would sync the data to Redshift or BigQuery.

12:33 Be- We because we are on ClickHouse Cloud, we are able to use ClickPipes. So, this entire setup is seamless for us, you know? we don't have to worry about setting up anything, you know? No Debezium required, no Kafka required, no Spark or Flink jobs required. And because in ClickHouse Cloud the data actually remains in decoupled in S3 kind of system. So, we are able to scale up and scale down our database and reduce the cost quite a lot.

13:03 When it comes to querying it, there we use ClickHouse MCP. And we have the Servo agents, which is the same Indus or Co-work that you I was talking about. So, we use Servo agents to do analytics on top of this data. And then apart from that, we have Redash and Metabase. There again, I don't think we build any dashboards. we use Servo agents to build dashboards on top of Redash and Metabase. So, yeah. That's it. Thanks.