PREVIEW Lightspeed 27 07 26 Fireside Chat

0:00 My name is Rohil Bagga. I've been with Lightspeed for almost seven years now. I co lead our early stage AI investing practice based here in Bangalore. And today I have the privilege of hosting Gaurav, Shweta and Ashwin. I think this panel is very unique because everyone brings a very different perspective and it'll be great to hear everyone's insights on open source versus frontier models. I think that's the broader framing for today's panel discussion. We'd love to keep it interactive and engaging.

0:31 So we'll probably do about 25, 30 minutes of of Q A. But then please do chime in and and contribute to the discussion before we turn it to our panelists. Just a quick show of hands. How many founders do we have in in the audience? Oh wow. Okay. Quite a few. How many operators? Startup operators.

0:51 Okay.

0:53 Super cool.

0:54 So yeah, this session is meant to be useful for all of you. So please do chime in with questions. Maybe we can start with a quick round of intros. Gaurav, if you'd want to kick it off.

1:06 Hey everyone. My name is Gaurav. I am from databricks. Oh, sorry, I have to press it once. It's fine. Okay. Is it working? Okay. Okay, now it's. Yeah. Hi. Hi everyone. Okay, now it's loud. My name is Gaurav. I am VP of Engineering at Databricks. I run our few products including you probably heard of genie, which is our first party agent.

1:43 Notebooks, which is our data science product for data scientists to use databricks, search, discovery, things like that. And this is. I've been in databricks for a couple of years now. Prior to that I have a lot of consumer experience. I also help kind of lead and help build the India site. I'm based out of California. Excited to be here.

2:10 Thank you, Gaurav. Hi everyone. My name is Shweta Rajpal Kohli and I, a founder myself founded Startup policy forum about 18 months ago. We're an industry platform that now represents 75 of India's leading high growth startups that could be together valued over $100 billion. Our main aim is to do policy advocacy on behalf of startups. So bridge the gap between founders and policymakers. We also do a lot of community building as well as help founders with global expansion and thought leadership.

2:46 A quick background. I've spent close to three decades between media and public policy, public policy roles at Uber Salesforce and Sequoia capital. That's now big 15.

2:59 Hi everyone, my name is Ashwin, one of the co Founders of xflo. We build cross border payments infrastructure. We operate in a very boring space. I mean that in a very, very non pejorative way. I've been building XFLO now along with my co founders for about five years. Before that I was at Stripe, build payments infrastructure there for another five years. Before that at Amazon again built payments infrastructure. I'm sure you're starting to see a theme emerge.

3:26 Before that I did a little bit of consulting in US and in India and in between. That is probably the most interesting part of my life. I taught in a public school in India, grades three and four for two years. So yeah, that's a little bit about me. Very excited to get to know all of you a little bit better. Also, this is an unsolicited plug. Shweta had nothing to do with this. We're actually one of the very happy, sort of like partners of Startup Policy Forum.

3:49 We've been interacting with them since December, December last year. Shweta didn't tell us when she started up, so this is her fault. Otherwise we would have just been partnering with her even earlier.

4:01 Also an unsolicited plug from my side. We're fortunate to be investors in xflow. We were the founding investors from the India fund and investors in Databricks from the US fund. So it's been great to work alongside Ashwin from the India Fund. But maybe Gaurav, we'll kick it off with you. Databricks has a Unity catalog product that is a very powerful gateway across open source and closed source models. Just give us a State of the Union from the enterprise perspective.

4:34 Are you seeing a shift of workloads from frontier closed source models to open source and if so, for what use cases? What are enterprises? How's the demand pattern from enterprises changing? Just give us a sense of that.

4:47 Yeah, great question. So I think it's sort of mirroring the evolution of the models across knowledge use cases. So this is like pre agents. So I think of it as pre agents and post agents. Pre agents, like knowledge extraction tasks. Those are very well handled by open source. Coin models have been used quite heavily for that. Similarly, we have followed that pattern as well.

5:21 Most of our knowledge extraction type work is on open source code completion, not agentic. Like the pre agentic world where you're trying to predict the next keystroke for code. That's also like open source is quite good at it. On the agentic side, I think the transition is starting to happen now because the powerful open weight agentic models were not there till About a month back with the arrival of GLM 5.2 and QME 3.

5:56 So that's starting to change. And so I think that's sort of in that transition phase and we'll see more and more action happening on the agentix side and the move towards the open source models. We are in like active, we're actively working on this. So I think for 60% of our agentic workloads we think we can shift pretty easily. I think it's identifying which workload to shift or which question to shift to open source is the harder problem and

6:30 picking up from there. Ashwin, is there a framework from a founder perspective on what workloads you should immediately shift to open source models versus stick on with the frontier models for.

6:43 Yeah, it's a really fair question folks. Just like before I answer that question right, let me just ground you all in context in terms of just like just a little bit about my companies because I don't want to give some advice and then that advice just doesn't seamlessly carry over. So we're a reasonably small company, we're like 100 people. And just to give you sort of like a headcount split, right, where about we're about 40% product engineering design, we're about 30% sales marketing ops and then we have finance and compliance and risk and partnerships and so on.

7:20 We're regulated in India, we hold a payments license in India, we hold a payments license in Gibbsity, we're a principal member in Visa. So all in all we're a very sort of like infrastructure, sort of like layer company. A good way to maybe think about us is we build Amazon web Services equivalent in the world of payments. That's what we do. And so from our perspective we don't really think open source versus like open weight versus close weight.

7:55 That's not the first thing that flashes in front of me. Right. Like the kind of questions we ask is very workload by workload dependent. The kind of questions we ask is

8:05 hey,

8:07 what's the cost to getting this answer wrong? If you get a regulated compliance decision wrong, the cost of that can actually be reasonably high because as a company we get audited by regulators every six months. We regulate it in different places. So it may sound like what's the big deal with two sort of regulations, but just to give you a sense like 100% company, we get audited by the Reserve bank of India, by IFSKA and Gibbsity, by Fintrac in Canada by four partner banks.

8:47 And all of this is like six Somewhere between four times a year to maybe like twice a year. So that's a bunch of audits. As you can think of getting regulated decisions wrong, there are real consequences. The next question that we ask is how sensitive is the data that I'm giving the model access to? So again, if this is, I'm just making things up mostly, but there's marketing data, okay, maybe that's not as sensitive as like transaction data.

9:26 Very different, right? And then obviously the third thing is just cost, right? We do think about sort of like cost over here. If you take these three things together, this is what naturally sort of like helps us arrive at, hey, should we go sort of like more open weight? Should we go close weight? If you're going close weight, should we go like Pareto Frontier? Should we go frontier? Or maybe something like more run of the mill, right?

9:51 That's the way we've learned to reason about it. This is not something that we've arrived at today. By the way. If you had asked me like 12 months ago, I would have given you a very cookie cutter, man. Don't even worry about it. Just go close to it and just go peredo frontier as much as you want. But as cost starts to become a consideration, as people start to discover their token maxing self, then I think these sorts of things become a little bit more.

10:17 More important.

10:18 And Ashwin, can you make it a bit real for us? Is there a workload that you were earlier using a frontier model for which you switched to open source or the other way around?

10:26 Yeah. No, It's a great question. So I'll give you a classic example, right? So actually maybe like 12 months ago is such a bad example, a bad idea, because 12 months ago my standard answer would be just use the latest frontier model Force. Let's not think that hard about this, right? But now we think much more deeply. So like just as an example, right, One of the things that we've been doing, we actually have an agent out there that actually crawls certain sites.

10:57 Let's take an example. Reddit. Such a great example. You crawl Reddit and if someone mentions a specific set of keywords, the agent then essentially identifies who this individual is, identifies the company of this individual, identifies, if this is a valid use case for us, then identifies the stakeholders in said company and then essentially starts firing off like a cold outreach sequence. And over a period of time, the agent, we've essentially given it enough intelligence to figure out who in XLO should be the right person to even reach out to these individuals.

11:36 Obviously, if you are connected to this individual or, you know, someone who's connected to this individual, you just have a higher rate of sort of like getting a response. Right. So we actually used to use like six months ago, we would use like a Pareto Frontier sort of like, model. Interestingly, Grok was just like, the best. I don't know why, but, you know, it is just like the best at getting us just responses. And remember, that's how we would measure.

12:00 Right.

12:00 We were like, what is the response that I am getting? But now we are still playing around with Kimike 3. But the Quinn set of models have actually been pretty good for us. They are multimodal. They're actually reasonably low cost. So I think we've actually literally switched over Romal because we've realized there's no point running a crazy expensive model on something like this. So when we started out and we were still shaping what we wanted, it made sense to kind of like, start with something that was a lot more powerful and more expensive.

12:40 But I think like, over a period of time, we've just discovered what works better.

12:43 And the previous framework was very powerful. Just to recap, you said cost of messing up data privacy and cost are

12:53 the three sort of. Yeah, these. The old three things.

12:55 Shweta, coming to you, I think it's really interesting to get your perspective on the policy around open source models, especially. I don't know if folks had the chance to read the Nvidia letter that Jensen Huang published. And there's a lot of chatter in the US on, hey, we need to protect against Chinese open source models. And the counter to that is this is the way we democratize AI. Right. And so give us a sense of the state of the union from your perspective, Shweta, on this whole closed source versus open source debate.

13:28 And where do you fall on the debate? Which side?

13:33 Yeah. Thank you, Rohil. I think all founders in the room would have heard this phrase very often, which is, it's a great time to build. It's a great time to be a founder. It's a great time to be a builder. I would say it's a fantastic, unprecedented time to be a policy professional right now. But you know what? It's a terrible time to be a policymaker or a regulator, because I do not envy the people out there who are making rules at a time when, before you can blink, you have a new model or the technology has probably leapfrogged miles ahead.

14:06 But on a more serious note, it is very, very daunting right now for policymakers. And regulators to take positions at a time when things are changing so quickly. And, you know, I think for India, we've always said that policies will be the India way. So when Europe got gdpr, India said, we will take our own sweet time and we will do it our way. And now finally, after over a decade, we have a data privacy law in place, which is something that will be implemented and brought into actual practice very soon.

14:42 But on AI, very specifically, we have the India AI mission. And you know what makes the job really hard is that the same body is supposed to promote the adoption of AI, is supposed to promote innovation. And we know the pressure right now and oh, has India missed the bus? And oh, things are very slow. And you know, India is not doing enough on AI, the government's not doing enough on AI, there's not enough innovation. All of that conversation in the backdrop of what's happening in the US and China.

15:12 And yet this is also a body that's supposed to regulate AI. So you imagine the twin objectives they're dealing with on a daily basis because you're supposed to say, oh, you know what, there are 10 things that can go wrong, but at the same time we can't be slow. So what is India's policy on AI? I think it's a three pronged approach. And while they haven't spelled it out in as many words, I think in our conversations almost every day with policymakers, this is what we pick up number one, innovation first.

15:42 And that's where the open source, open weight versus, I think frontier debate comes in. Because if you're innovation first, you naturally cannot take a position where you are seen to be banning or restricting in any way. So if you're innovation first, you're likely to be in the camp of promoting open weight. I think the second is light touch. So the government's very clear that yes, we will regulate, but we want to keep regulations light touch. And the third is legislate when needed.

16:15 And that's a very important one, which leaves the room open for the government to say, you know what, this is where we see risks. And now we step in. Now, when will that moment come for India? The position's very, very tough, given what on one hand, you want all founders in the room to have access to the best of technology, to the best of models, and whether they're coming from the US or from. I don't think the government wants to restrict innovation of any kind because if you need to build on top of them, you need to build on top of them.

16:43 And that's the reason why you're not seeing any overt statements, despite all the concern that we've seen around data flows and data localization and the noise around data in a pre AI era. At the same time, we know that when the government needs to step in and when national interest is paramount. We saw what happened with gaming, right? And at that time, no logic prevails, right? No amount of industry first or promote innovation or this is great, or X billion dollars of investments coming in.

17:17 Oh, this is a great sector or jobs or revenues or taxes. None of those, none of those arguments hold good when it comes to is this in national interest? So I would say it's a bit of a wait and watch policy legislate as needed. As of right now, we know there is, there is, there are no restrictions, but it's a lot of wait and watch. The only thing I could share here is that India so far has taken a position that India will not have an overarching AI law.

17:44 Like we will not have an AI regulator because we believe that the current laws, whether it's the IT law or the data protection law, the DPDP act along with many other laws together take care of the concerns around AI. But like I said, positions change.

17:59 Yeah, you answered my next question briefly, but is there anything that founders in the room should do differently over the next 12 months to ensure that they're moving in accordance with how policies around India AI are shifting? And you might have a perspective based on all the startups that are part of the policy forum, but any tips or words of wisdom for founders to ensure that they move lockstep with policy?

18:28 So I think the one thing that I would say is please, you know, share your experiences, share what you need from the government, engage more. You know, there is such a fierce psychosis around engaging with the government and that's part of what we do, I think. Thank you, Ashwin, for that plug. But I can tell you, you know, XFLO is, you know, I didn't know Ashwin and gang like I think six months ago and they have been so active and engaging on our government meetings like we.

18:59 And honestly, at some point maybe Ashwin will talk to it. But you know, the fact is that in today's times, with geopolitical uncertainty, with technology uncertainty, if we are not engaging with policymakers and telling them where your roadblocks are, is it compute? Is it capital? Is it. You want the policy to be open weight, like if the government's not going to hear from builders, we're not Going to get policies that are meant for you. And this is the time the government's very open to listening.

19:30 So please engage, please be out there, please be a part of the policy making process. Because you know what? The government doesn't know better but they are here to support and to do what is right. But sometimes, you know, knee jerk policy decisions happen because as I said, it seems like the best answer at that point.

19:50 Super helpful coming back to you Gaurav. I think the biggest, I don't want to use the word issue, but the biggest challenge that people have with open source models is that you have to think about a lot of things beyond the model as a team and as a founder, whether it's the harness, the scaffolding, observability monitoring. So any insights on how enterprises or startups are managing those aspects of using open source models, Any best practices, any tips or advice to founders on how to think about all these building blocks that are required to take a model from let's say POC to production?

20:31 Yeah, that's a great question. I think obviously if you are using one of the close frontier models, you get an API, you can sort of trust them, they are handling everything for you. So I think it's just, probably just evolution, right? Like the open weight serving is a little bit behind and whether it's hyperscalers or US companies like us, or even the new clouds, they're going to solve it over time and at least the way we have done it is like a single place where there is clear governance on the usage, whether it's who can use what, how much can they spend, things like that.

21:12 Also where can the request go like that is generally the biggest worry for enterprises that where will this request go? So we like the transparency on that is evolving. Like can it cross India's boundaries, can it go to Singapore, can it go to US in Europe? If you are in certain countries, it can go to certain other countries. So those are the challenges that are being solved. We solved it internally and now we're solving it through our products.

21:38 So I think that's one thing. And then, sorry, governance, transparency and then observability as you said. I think you'll see like the cloud, the hyperscalers or some of the data AI companies like ours, they will basically become multi AI providers. There'll be a single API, single place, you'll have the cost controls, governance controls and so on and then you can use it with less of that trepidation and fear. I think when you are doing it on your own Then it is a little bit more daunting.

22:12 Got it.

22:12 Okay.

22:13 And does databricks have a product that solves some of these components for you? Want to talk a little bit about that? Yeah.

22:22 So we have a Unity AI gateway. So that's for governance and cost controls. And then our FM API provides access to pretty much all the models at this point of time. So whether you want to use even the closed source models, so even the OpenAI or anthropic models, we get access to them as soon as they're released. And then we serve the open source models or open weight models through as many data like through as many aws, GCP or Azure options as we could procure, like the cloud, procure GPUs and compute from them.

23:01 We try to serve it.

23:02 Got it. Okay, Ashwin, coming back to you, something that even as investors that we hear a lot about is founders have this fear of vendor lock in, especially as it relates to committing to a model or a model provider. Just curious, from your perspective, is that quote unquote fear real? Is that overstated? Should that be a factor when people think about choosing across the menu of different model providers?

23:37 It's a really good question. And I think the question isn't so much even related to even AI in any way. I'll give you a simple example. When we were at Stripe, when I was at Stripe, we used to the data shows that if you have a merchant and the merchant is using Stripe for pure payments, when the merchant is up for renewal, say in the next one year or two years and so on, just pure payments is actually reasonably replaceable.

24:13 Ok, what's a good way. So think about payments as the equivalent of storage and compute on AWS, just S3 and EC2. And tomorrow if you wanted to rip and replace your storage and compute, be sure it's work, but it's not that hard. But now all of a sudden, if you're using DynamoDB on AWS now you'll think twice because now you have database lock in. Now if you're using say auth, okay, now you'll think even more. If you're using say cloudflare for reporting, you'll think a little bit more.

24:42 And so our philosophy has always been that we don't really want sort of like very tight vendor lock in. And the way we sort of ensure we don't have tight vendor lock in is we'll get the equivalent of the S3 and EC2 and even on AI. And the equivalent of that, to be very honest, is there are Many enough and more tools out there. Gaurav actually described what databricks is doing here as well. Engineers will always want their harnesses, there will always be some observability, there will always be some data sandboxing, those sorts of things.

25:18 But I don't think we think model backwards. That's not our jam. Right? Like there are things that we are much more closer to. Like for example, over a period of time we've just scaled to using more and more AWS products. So is it impossible for us to move from AWS to gcp? Yes, but the opportunity cost is just way too high for us. Right? And so from our perspective, do we think about vendor lock in? The answer is yes.

25:46 Yes. And the way we sort of like the way we manage vendor lock in is we think reasonably hard, right? About when we actually roll something out company wide. You want to do something like as an experiment, as a two man crew, by all means go crazy, right? But the minute something becomes institutionalized. And by the way, this is a learning for me as a founder, this is my first time. By the way folks, rights have only been added five years, right?

26:14 But it's incredible how like with just 100 person company, it's not that big, right? Maybe the 100% company, if I say yes, you know this zoom meeting thing, this is too expensive. I think we're going to use Google Meets from now on. Best of luck. Seriously, best of luck getting people to just switch that over. Now if you're talking about like a harness which like people actually like have feelings for.

26:37 Right?

26:37 Like I love this hardness, right. You can't take it away from me. Yeah, best of luck.

26:43 I like that. And for founders in regulated spaces, any hard won advice on architecting your tech stack? Any learnings over the last five years specifically on like founders operating, let's say healthcare, financial services.

27:03 Yeah, I'll stay away from healthcare. I don't know the space very well so I will not comment on that. On the financial services space. Yeah. I mean folks, if anyone's building like in the regulated space and you all want some advice, please come and chat. I think the regulated space is actually pretty small anyway. Something that what Shweta said, which is I think the way we partner with Startup Policy Forum is we're now thinking a little bit further out.

27:28 We're trying to shape policy through input. But then there's also policy that already exists today. There's regulation that already exists today. Our number one learning is don't try to do something experimental in a really tightly regulated space just don't do may sound cool, it probably is cool. But cool is not what you get rewarded for. What you get rewarded for is if I tell my merchant, hey, Listen, you have $98.34 in your bank account.

28:02 He better have $98.34 in your bank account. If you're off by a cent, very bad things will happen. You will lose trust. Right? And so from our perspective, correctness, there are two things, right, that kills infrastructure companies super fast.

28:18 One is

28:20 if you're off from a correctness perspective, and two, if you have any kind of security breach. And so in these spaces, running sandboxes is a really good thing. Testing out whatever you're doing is a really good thing. Testing for correctness is a really good thing. So I think our advantage is that very frankly, I'm 44, I'm not AI native.

28:51 I didn't grow up in this space. I grew into this space. And so I'm like a Web2 native, I suppose, right? So, yeah, just slow and steady.

29:02 Awesome. Last question to you, Shweta, before we open up to the audience. Policy and policy making seems to be like a bit of a black box, right? And then one fine day, the government sends out regulation and everyone must abide by it. And you touched upon this in your previous answer as well. But any case studies that come to mind, and doesn't have to be AI specific, but could be with the data policy, with any specific industry or software more broadly, where founders worked closely with the government, help shape narrative, help shape policy, or work closely with you, any case studies that you could give us?

29:40 Absolutely. I can go on and on, but I'll try to keep it brief. Rohil, I think what we often don't realize is that there is a science and there is an art, and it's a combination of two, right? So there is what we call a clear policy toolkit available. And what is that toolkit? I mean, intuitively think of it. There are consultations, there are submissions, there are direct meetings. Advocacy is a very, very detailed and a very scientific way in which you can influence policy.

30:10 And a lot of it is evidence based and data based. To quote an example where we worked very closely with founders, and I take a lot of pride in this one policy that we truly moved was, as many of us know, that domiciling overseas was a thing a lot more even for the generation of founders that are now going ipo. And when a lot of the companies, including Grow Pine Labs, Razorpay, many others realized that you Know what?

30:38 Now they want to IPO in India. They had to reverse flip, as we call it, or re domicile. Right. I remember working very closely at that time with some of these companies and we started seeing the pain points that founders were facing when they were reverse flipping. And one of the biggest pain points was that in Indian law, under the Companies Act, a foreign to Indian merger had to go through a judicial process called the nclt, which is the National Company Law Tribunal.

31:06 Right. And that itself meant that in typical Indian, I would say judicial process, it's Tariq Pitarik. So you are just lost in this sort of judicial process that could stretch between 18 to 24 months. Now imagine 18 to 24 months in the life of a founder. Your valuations will change, the market, sentiment will change. The capital markets are not looking as exciting as they did say. And so when we realized that problem, and I remember going to the Indian government, to the company's, the MCA secretary, the Ministry of Corporate affairs secretary, and he obviously did not know about this problem.

31:47 And he said, oh. And I remember just walking to his room and saying, you know what, we have a bunch of companies that are trying to reverse flip and these are the pain points. He said, can you get the founders to come and see me? And I remember I phoned Lalit of Grow Harshal of Razorpay and they decided to just join this meeting. And I said, and the secretary said, let's meet next week. So we had that date locked in.

32:07 They flew down from Bangalore to Delhi. We sat in a room and he literally took out the company's act and we had that thick book with us and we went through all the sections in the company's act to say, what's going on, what's the way out? And together as we sat through, along with Gross cfo, I know there was Meeshu and Pine Labs also in the room, their teams. And we came up with a solution where we said, you know what, there is a provision to fast track.

32:34 And in India, a merger within India is allowed in a fast track way where you can buy, bypass NCLT and you don't need to go through NCLT and you know, but that requires a change which at that time felt like it would be a legislative change, which meant it would need to go to Parliament, an amendment would need to happen in the Companies Act. But then the secretary, along with his team of bureaucrats said, you know what, I think we can do it through a notification.

32:59 It doesn't need a parliamentary change. But it took consistent advocacy from us for an entire year because it needed inter ministerial approval. So we knocked the doors of Finance Ministry within Finance Ministry, knowing you have to go to dfs, you have to go to dea, you have to go to the Department of Revenue, you have to go to every, every department. And then we had to go to Invest India, we had to go to the Corporate Affairs Ministry, we went to DPIIT that manages startups and built this inter ministerial understanding of why this is so critical.

33:33 And how did you do that? Just to quickly wrap it up, you have to say, okay, in the next two years, 40 companies are gonna reverse flip. What does this mean for India? How will this impact jobs? How will this impact your capital markets? So unless you're providing data to the government on why this is good for India, not why this is good for grow or not why this is good for Pine Labs, but why is this good for India once the government sees that they're well meaning bureaucrats and it took an entire year, but now let me tell you, the impact of that grow took between 12 to 14 months because they had to go through the NCLT process.

34:06 They had already applied, so there was no way to scale back from it. Pine Labs had to go through the NCLT process. Both companies took, if I remember correctly, between 12 to 14 months to reverse flip. But Razorpay, that is now going to go IPO this year. Razorpay applied once. The policy had changed and now that policy meant you don't need a judicial process. And guess how long did Razorpay take to flip back? Two months. So that's the impact policy can have.

34:33 And all it took was just building that awareness within government stakeholders.

34:38 Yeah, super powerful.

34:40 Sharing that. I have lots of questions, but I'll pause here and inviting folks from the audience to chime in. Either of us would be happy to answer any questions that you may have. We can maybe do a show of hands if anyone has questions. Yeah, yeah, we, we'll come after.

35:04 Have a question for Gaurav and Ashwin. Both of you have phenomenal tech teams,

35:10 so you need to be a little louder.

35:12 Yeah, I think maybe like a year back the question of build versus buy was very much buy internally. And given that both of you have incredible tech teams, how has that changed? And you know what factors are maybe making you consider? Are you considering building SLMs or models within your own products?

35:34 Oh, just. Your question is just for the models or is it for in general software? Yeah, see, I think for us we have consciously decided that we are not building the models like at the frontier level, but that's like a decision we made. But frontier models is where you get the maximum learning. So all our cutting edge stuff happens there. We are extremely AI built.

36:06 We tell our developers, our teams to build using the best in class. But as I said earlier, as the products get mature, there is a lot of opportunity to either use something open source or fine tune it. In the past, what we found was that fine tuning wasn't that valuable. We did get immediate gains, so we sort of building it in house. But the next iteration of the model will subsume it. So you basically bought only like three, four months of improvements ahead and then the next iteration of the base model itself will kind of solve for it.

36:47 So I think we'll kind of continue to think of it this way unless the models start to saturate. When they start to saturate and you'll see in the performance, then there will be a lot more value in either building or fine tuning specific models. We are on the other hand definitely thinking about smart routing. So as we use multiple models, one of the things that is critical is when to send it to what. So it's not really a LLM model, but like, how do you do that?

37:15 Is a critical problem.

37:18 Such a really good answer. I think I have a bunch of things over here on my wall of shame, right? So maybe like six or seven months ago, right. So I'm an engineer by heart, right. So like mostly on the product and engineering side. So about like four or five of us, we were like, you know what would be really cool? We should build our own model. That's literally how we thought of things. And so then we were like, well, we can't obviously introduce it to on production, so we should test it internally.

37:48 So we said, okay, cool, we'll build a support bot. And when I say support bot, there's actually literally a bot like a slack bot in Xflow called Bazinga because we thought Bazinga is cool. And we, we trained our own model of this. We basically forked off, I think it was Quinn or Lama, I don't even remember now, to be honest. So we forked off one of these and we sort of trained it and we did a little bit of rl and at the time it actually worked pretty cool.

38:18 I thought we were like, wow, look at us, we're so smart. Good job. And then this new sort of Pareto frontier model comes out and just destroys Bazingar. And you can see this now, there's actually like, there's a Bazinga Classic and then there's Bazinga New because we couldn't unfortunately like you know, we like emotionally connected to Bazinga Classic so we couldn't retire it. And so but Bazinga New just like destroyed us. Right? Like so then we realized, okay, you know what, this is absolutely like a very bad use of our time.

38:50 It's better if we just like we are not a model shop and let's not pretend to be like we can do obviously cool things and there's nothing wrong in that. But what's cooler than doing Bazinga Classic is maybe getting a few extra hours of sleep. So yeah, we don't. I think we've realized we've tried it. Maybe we'll be better at it the next time. But I think like we just have other things to do. That's the honest answer.

39:19 But it was really cool stuff.

39:20 I and just to add on to that. So as I said, we did try it. Everyone initially goes to building models in school. Let's do it. Or even fine tuning or distilling. But we have learned like that's not worth it, at least right now as I explained earlier and as I think the action that we are spending the most time on is the hardness, context and memory. Those are the three places where we are spending the most amount of time using the frontier models.

39:49 And we have a good belief that over time we can move them over to either open source and there's plenty of time to fine tune and get a little bit more niche when needed. And then one more place we spend a lot of time on is, and this is actually quite important, building our eval sets. Because no amount of eval sets that you see in the wild are going to be perfect for your problem. They're all benchmarketing essentially.

40:18 They're all like creating this benchmark for marketing their own whatever they're cooking. So I would spend time on building those because that's how you would know like what are the actual problems in your product.

40:33 Got a question there? Yeah, just need to be a little loud.

40:38 Yeah.

40:40 Oh, we got a mic. Just.

40:50 Going back. Especially like data. Something of data.

41:09 Little louder.

41:10 Sorry.

41:10 Yeah, I was talking about data having like SLAs and agreements to these providers. Data reservation, all sort of garages around. What do you do when it comes to these open source. I think you mentioned something about taking a computer on this data.

41:34 Yeah,

41:40 Like, for example am right. We don't get. So we have a dollar arbitrage problem that if I take a lot credits and like going land Right. And we all know how OPUS performs and we run out of credits and then what you shift. Right. As a nation, we had a notification come on like and we don't have either a decentralized or even like localized provider on these open rate models. Right. Is there something that is being looked at?

42:12 I'm thinking both in terms of this coming out of the web. Three space and a lot of things are very deterministic.

42:28 So to your first question, we do run those weights. Yeah. So it's basically inference running on whichever. So I don't know how much you know about data databricks. Databricks basically works across three clouds. Azure, aws, gcp. So whichever cloud you are on, wherever your data is, we will. We kind of work on top of it. So whatever was true for your data and compute will be true for your LLM inference. Also, assuming it's open weight and it's being run on those clouds, we provide those guardrails.

43:01 It's actually not going to anyone else. It's running there. It's running in either Azure or AWS or gcp.

43:16 So over there it becomes a little like.

43:22 Yeah, so we haven't done any new clouds yet. But as we do it, I'm sure we'll have to think about that. We are also getting GPU constraints, so we'll have to use some of the new clouds but then we'll evolve accordingly. I think there was a second policy thing also.

43:37 Right.

43:38 I think you already answered it. But if you want to give more color else we can take it offline.

43:43 Yeah, sure. Happy to discuss it offline.

43:47 Last one and then we'll. We'll need to wrap.

43:50 Sorry.

43:50 We'll go.

44:03 Have increasing open rate models then popular cases. How does one decide what model can work? Have some confidence that you know this is less of the Fine.

44:23 Okay, but give some time that the cross are. Yeah, So I think as I said earlier, first of all you need to build your own eval sets. So whatever problem your product is trying to solve, you have unique requests that come to you, make sure you have your eval sets. And the first step would be just evaluating against the answers that you get from model A versus Model B.

44:56 That'll be the first step. That's how we do it. We build our eval sets. Then if you look at a bunch of blogs that we have published, they are based on actual usage versus benchmarks that people have put up. And if you're lucky and everything's better with OpenVid, you can just switch to it, but likely that's not going to be the case. You will see that on a certain slice of traffic, it works great. On certain slice of traffic, the other models are better.

45:22 So that is where a choice comes in and you have to make that choice. Yeah, but the eval set is hopefully not as costly because you're just generating a sample. It's not all your traffic. You generate a representative sample and then use that. So I think this is actually important to internalize.

45:54 It is not like these models are not deterministic. So the best you can do is take a large enough sample, at which point you can say, for 80% of my tasks, and you have to categorize them and so on. You can use model A versus Model

46:08 B. I'm so sorry, I'll interrupt. You guys can chat offline. We'll do one last one and then we'll wrap up. Sorry about that.

46:17 I would think this is the very appropriate question, but we have, we are very early stage startup and we are on our first.

46:26 We are on our talks with our first few design partners and something about

46:30 the laws and regulations are something that we are consistently failing upon. And we're not able to actually figure out what route or what we should actually do, you know, to actually make sure that this is something that is shippable to something large. So where would we, you know, like start? Should we start with a lawyer or how does that process go?

46:53 Actually happy to take that. So I think there is. Every founder should do what we call a stakeholder mapping and a regulation mapping before they begin. And to take forward Ashwin's point, you know, there was a phase in tech policy where it was actually cool to be operating in the grays. And you know what you're like, you know, I think I belong to that Uber era of tech policy as well, and many ex Uber folks here in the room as well.

47:18 And I think it was almost like, you know what? Go innovate. Regulation will follow and it's all right. And it was almost a cool thing to do because you'll be too big and then the government will have no choice but to regulate. I think that era is over. We're now at a phase where, you know what regulators and policymakers say, don't try to be cute with us. You know, operate. Don't operate in the grays because you will likely get shut down.

47:39 So I think it's very important that you start building in, you know, regulations and policies and there will be areas that will feel like they are grays and you don't have the answer. But I think it's very important to start engaging and understanding and sometimes it's also very important to know that there are multiple laws, multiple stakeholders. Now let me tell you, deep tech is such a catch all phrase right now. Everybody is so called building in deep tech.

48:02 And then there is also an intersection of deep tech with AI and you know, I mean, but everybody just says deep tech. You know, we're going to invest in deep tech, we're going to be building deep tech alone. If we start doing the stakeholder mapping of the government stakeholders that are involved. I mean, it's a list that will take us months to even get through meeting everybody in government for it because there are at least 10 ministries that are involved.

48:25 Does deep tech sit under, you know, Department of Science and Technology or under dpiit or under the, in the, under Metis? Or is it under the office of the principal Scientific Advisor to the government? When you start mapping it all out, you will realize or if it's robotics or aerospace or are you going to go to civil aviation or so it's very complex. But the right answer is you should know your laws. And sometimes policy can actually, you know, you won't realize it, but it can actually be a lever.

48:52 It can actually provide you those necessary tailwinds to, you know, actually, you know, work around them.

48:59 Yeah, actually I think it's very coincidental, but we started off as a deep tech startup and we pivoted into something more software and whatever you said made so much sense because we were on the bounds of regulating or creating so many regulations that it was much better

49:19 for us to shift or pivot than

49:21 to continue in the same space.

49:23 No, I would say don't let that be a barrier. There are sandboxes everywhere. Everybody loves sandboxes. So you have to find a way around it, but while being on the right side of law. So I would never tell a founder that give up or pivot because it's, you know, there are, there are ways around it.

49:40 It's almost like you're again on the grayscale, like very border to the gray scale. But then you're also not on the grayscale and it's more illegal maybe.

49:50 So

49:53 gaming.

49:54 Yeah, yeah. So I mean illegal is the extreme end. There are grays and there is green light. And right now most of who is building whatever, I mean, you could say so much of it is technically grays. But like, what I'm trying to say is that things that are obviously gray, there are ways around it, engage with Policymakers write to them. There are, there are so many ways in which you can protect yourself and yet build and yet be seen on the right side of law.

50:18 That's the part you need to do. But you can't stop innovating because, I mean, I'm not suggesting that, oh, if it is not written in law that it's fine, then you should not explore. Because by that logic, no innovation would happen because everything is in graze. In that sense,

50:35 as a founder, I

50:36 think it's a very good opportunity to just understand that this gray space is something that you can capitalize on to make sure that you're not a mis.

50:48 Yeah, we'll talk, we'll talk offline.

50:49 Yeah, awesome. I'll take the liberty to close this out. Last one, rapid fire. Put all three of you on the spot. One open source model that you've been very positively surprised with in terms of performance and that you'd personally bet on for the next 12 months, Quinn, and for what use case.

51:08 Okay, three use cases come to mind. The marketing agent that I mentioned earlier, we're now extending it so that it works across Reddit and LinkedIn and email and so on. And it's actually been surprisingly good, which shocks us a little bit. Two payments means payments fraud. So fraud is like a big deal for us. So what's interesting is it's a lightspeed funded company called Alloy and we use them for transaction monitoring and Quinn is actually beating them on certain benchmarks, which gives me pause, which gives me a lot of volume.

51:51 It gives me pause.

51:52 Right.

51:53 Which is insane for us. And so it's just that we've been able to train Quinn on very sharp domain specific data and it's just getting better and better. And I think that's been pretty surprising for us. And the third use case, very interestingly is onboarding. So we have to onboard users, we have to do KYC on them and KYB on them and so on. And what Gwen's been very good at discovering is inconsistencies during onboarding.

52:31 And so let me show you like an example of inconsistency. So imagine someone that is trying to onboard themselves from Punjab and they give a GST and their GST ID is in the state of Karnataka and the GST was created before the company identification number was created. It's starting to flag those sorts of inconsistencies for us, which honestly did not even strike us.

53:02 So we're like, wait a minute, right? Like, what's going on over here? They're actually challenging, like the model and the model behavior is starting to challenge, like our seasoned risk analysts, which is pretty stunning.

53:19 I'm not the tech expert here, so not the perfect one. But, you know, we do pose that question to many founders and I think the only answer I would give to that is that the answer changes so dramatically. I would say every fortnight, you know, and we were running this series called AI in Action and you know, we started exactly a year ago and if you go back to the founders and say this is what you said.

53:41 So, you know, we also do a rapid fire and sometimes we say, okay, if there is one app you would want to keep and if you had to delete everything else. It's some fun questions. It's shocking that they're like, oh my God, we can't believe we said that. Can you not run it? Because, you know, now we've totally changed our answer. So I would say it's changing so dramatically that I don't know if he will have the same answer, you know, three months later.

54:03 Yeah. So just to build on that, I have a two part answer to that. We are actually truly multi AI. So there's no answer is, I guess no single model. But as I keep coming back to evals. Right. So what we have focused on is building eval sets for real use cases. So anytime a new model comes out, whether it's in the sandbox or for real, we can quickly evaluate it on those use cases across quality, of course, how good it is, cost and latency, all three, because all three are responsible for the ultimate user experience.

54:39 So I guess we will switch to whichever one works at the moment. Right now I would say if you just ask me in point in time, it's either glm, Kimi. We were very bullish on glm, but Kimi just came out last week, so it changed. It can change. GLM is also doing quite well.

54:57 Awesome.

54:58 Cool. With that we'll wrap the panel. All four of us are here so you can find any of us if you have follow up questions. We also have refreshments for everyone outside. So please do stick along and feel free to come up to us to ask any question that you may have. So thank you so much and thank you, all three of you for.

55:19 Thank you. Thanks, dude. Yeah,