Sunita Verma - Rate Limiter on AI Adoption Is Organizational

0:02 Good morning everyone. I hope you had a wonderful time yesterday, enjoyed the sessions. my name is Sunita Verma. I am CTO at Ironclad. prior to coming at Ironclad I was I spent almost 18 years at Google building and deploying AI technology at scale for various use cases. so I'm going to talk about I'm going to make a claim and I'm going to be use the next 10 to 15 minutes validating or supporting that claim with some of the data that we have seen. And my claim is that and actually the video that was playing beforehand sort of set it up properly very appropriately for me. The adoption of AI in enterprises is not stalling because the technology is not working or is not there. It's stalling because of other reasons. And one of the other reasons tends to be processes, culture and so on. And the question I think we all want to ask is why should you care?

1:05 Why do you want to care about whether the technology gets deployed in an enterprise or not? And my claim to that is all the good work, all the great work that we heard about yesterday that we're going to hear about today needs to deliver value. Otherwise, it's all for a knot. So the who are the eventual users of this technology, it's going to be all the people who live in thousands and thousands of companies that exist around us.

1:37 So I want to start with a story. nine months ago which is approximately like a month or two after I had joined Ironclad we were having a discussion about features and I deliberately ran an experiment there. I asked two different engineers for an estimate of what it'll take to build that feature and guess what I found? I found such diverging amount of time in estimates that it gave me a pause. I'm like what's going on here? And the when I dug into it, one of the things I realized it's actually less about the seniority of the engineer or familiarity of the engineer with the codebase. The difference was one of the engineer was AI pled and understood how to use the technology and work with the technology and steer the technology and so on. And the other was using AI technology mainly as a chatbot but felt that they already had mastered the technology.

2:34 And so I had a choice to make at that point. I knew that the rate at which the technology is advancing, this gap is going to get compounded between people and I could actually accept to have a twospeed engineering organization or I could do something about it. So one of the things I stepped back and thought about is why is it that people feel that they have mastered the technology when they actually haven't and at least my assertion is in the previous technologies whether it's mobile whether it's cloud when you had an error the error was obvious to you the technology sort of pushed back and you knew you had to fix the error you knew you had to learn what's going on or what's going wrong before you could make progress but with AI Because the interface is natural language.

3:28 Just by talking to it, you sort of feel you already know what you're doing without realizing that the outputs need to be validated. The outcome needs to be validated. So it actually just it's a very different mental model that you have to use with this technology. And yes, many of you sitting in the audience may say, "Oh, I already knew that." But there is a huge workforce out there that is not in the same mindset and a lot of people sitting here. So it's if you want the technology to be adopted by that set of people it's important that we do something about it.

4:07 So one of the things I also learned from lot of other companies talking to a lot of other companies is and these are big fortune 500 sometimes fortuneund companies most of the companies say oh we'll enable our engineers and I think that's a myopic view to take because you enable one function but then you create bottlenecks in other part of your organization so one of the very deliberate thing we did at ironclad right from the beginning is we will enable everyone at the same time yes some people were already pretty AI pled and they felt that they were not they didn't need this but we were very clear that we want engineering product and design to work together to get enabled together so one of the goals we set for oursel I set for the team is we want to make sure we are a company where building with AI and building AI is not a bespoke act for certain people everybody can actually build it everybody can build with the technology So three principles we basically said there aren't just going to be a you know halo set of people on this side that are doing AI and then they pull everybody along with them that hard parts of making sure the outcome validation the eval the eval harnesses everything is going to be through tooling so that so that everyone gets the benefit of it at the same time and then it's not going to be more like hey only certain top people get to use AI AI and nobody else everybody has to use AI and everybody will use AI. So we basically started with these principles and what I did was I leveraged one of the pro things we had coming up. So I was just three months into the company at that time. We had a hackathon coming up in December 2025. So I'm talking eight months back which is an eternity in AI. so December 2025 we had a hackathon coming up and I wanted to make sure that the team actually comes prepared with AI in this hackathon and we get we basically see AI build related outcomes from the hackathon. So I created a 20-day program prior to the hackathon that we ran prior to the hackathon to basically do the skill building across the organization and we are at a much different place as a company now but this is basically eight months back day one because I built technology I'm very familiar with the technology I actually took classes for the whole team for and I think we allowed it anybody across the company to be able to join those classes. So I took classes on basically LLM fundamentals, how LLM works under the hood, how to think about context, incontext learning, prompt optimization, tuning, evaluations and so on. And then for the next 20 days, for rest of the two to 20 days, one engineer every day will basically come and talk about how they built with what they built with the technology, how they built with the technology, how the what problems they ran into and what learnings they took away from it. And we kept the sort of the momentum going through slack channels, through interactions, through you know sort of impromptu gettogethers that engineers wanted to do among out there themselves.

7:18 And so we ran the whole program in 20 days and come hackathon we actually got like huge amount of participation. We got lot of people building capabilities building new things that I didn't think that the team was basically team could build but they surprised me significantly on a positive side. So fluency wise we were there as a company at that point and one of the things I was expecting to see out of it and this is now we are talking about January 2026 again still 7 to 8 months away from where we are today. So I'm just telling you a story this is not the state of the company where we are today. January 2026 the the the sort of like the feature development part didn't sort of take off as as at the same rate as I had thought.

8:07 And again I'm like okay what happened? I thought we had already upskilled everybody. I should be seeing we should be seeing like all this building happening in the product. In in in experimentation we were already there but in the product we should be seeing all this building happening. So what's what's what's going wrong and the and I realized it is actually Andel's law and what Andel's law says is you update one part of the process but now the parts of the process that you didn't touch actually become the bottleneck and one of the things I realized is yes we were building every function was building AI and building with AI but we hadn't we had not shifted ed the process at all and we were still following a waterfall model of development where product will do its research they'll hand it off to UX and then it'll come to the engineer and engineer will basically get done really quickly and then most of the time they'll be just waiting for answers from the product team I'm like oh okay so this is actually where we need to fix the the we need to actually change our approach so we basically tweaked the process we I already talked about education being the infrastructure And part of that education being the infrastructure is building a shared lexicon, building shared tooling, building harnesses and making sure that there are common systems for context management, rag, memory across the board. So we are basically brought all the organizational memory into a single system. But I think the it's the process part of the shifting that actually was very interesting. So what we do now is when trying to build something the relevant people will come together early on they'll basically get together they'll discuss the problem they'll debate the problem they'll come up with a solution they'll iterate on the solution and they will memorialize the solution as either a markdown or a notion page and I don't know if any of you attended professor Ian's talk yesterday one of the things he was talking about is why the outcomes from AI are not what people expected it to be. Part of it is under specification of the requirements and having everybody come together in the early stage of the process allows us to flesh out those requirements right in the beginning before handing off that fleshed out work to an agent to execute to an AI agent to execute. I'm not saying we are perfect there but I think we have seen the benefit of specifying like clearly specifying the constraints and the boundaries and the solution space of the problem before giving it to an agent to execute and we are seeing really good outcome and I'll show you some data around that as well.

11:04 The second the last but not the least thing is basically we have created tools and guardrails that ensure so obviously there are code review agents but there are just lot of skills that we have built to ensure that the quality of the outcome that is produced by these agents especially on the coding side meets the requirements the standards that we have in our company and so that nobody has to wonder whether their agent is doing the right thing or respond.

11:34 So where are we as a company? so what I told you was up to January 2026 AI space moves really fast and we move really fast. We moved really fast to to keep pace with it or to outpace it sometimes actually. So we have created an agent framework and this is sort of a cartoon diagram. I I omitted some of the deep technical details from here. this we have created an agent framework which basically enables everyone to anyone to create an agent really quickly and it's based on agent SDK from OpenAI and AI SDK from Versel connected to all the frontier models we we have also started tuning actually our own models so some of those models are also getting hooked up behind these behind these agents and we built all the other infrastructure around it for continuous context optimization memory management ment continuous evas prompting and and so on. So this is the this is sort of the framework that the whole company uses now to create agents.

12:39 We've also basically create automated evals in large parts of our system. and eval tend to be one of those underspecified things or under invested things that people use. So we do have LLM judges that are actually evaluating the output that is being produced by a lot of these agents and then they give it to an evaluation agent which then compares it against a rubric and till it's till it basically decides that yes the eval is actually coming very close to the rubric. It'll keep running those continuous evals without involving any of our engineers in the process. So once the engineer once after something has been built, reviewed and checked in, it basically goes through this evaluation process automatically.

13:23 We've also built our own code generation harness. We evaluated everything actually out there. We basically evaluated Codex harness. We evaluated Claude's hardness. And so what we did was none of them were coming close to what we wanted to build. So we basically took them as benchmarks some of the open source ones as sort of the substrate and we built our own code generation harness on top of it. we are using temporal and temporal was here yesterday so shout out.

13:51 so what people now do and by the way our product managers are also coding now because it's just so easy to do the right thing. so anybody can actually interface with this harness. they can actually give it a job to do. whether it is a UI that you want build, whether it's a backend part of the service that you want build, whether it is whether it is any other part of the code you want written, you can give it to the harness. The harness actually spins up a VM. It basically sets up the whole environment almost equivalent to a production environment in that VM, generates the code in that VM, tests the code in that VM and produces a review ready PR that can then get reviewed by either humans or a coding a review agent.

14:37 This is sort of what the web interface for our harness looks like. I've deliberately sort of made the colors weaker because there are some things that I didn't want to do that are more proprietary to us. So basically this is a hardness that people use. It's in dog food right now and we are getting really really good results from it. we've also delivered a lot of value in the product. By the way, as I said, we are a legal tech, commercial legal product. Almost all the companies in Silicon Valley are customers, including the Frontier Labs. And so there are a lot of products that are that are agents that live in our product today. And those agents are act like turning real running real contracts which are billions of dollars of value that is running through them at this point. nine months later, this is where we are like it's a very different place for me as well as as a company.

15:25 these are more engineering statistics and by the way the drops you see at the end of the curve is incomplete data. It's not that you know we've suddenly become we are suddenly back to square one kind of thing. So our execution velocity is just very very different than what I saw a year ago. Just one last thing I want to leave you with it's not that this is something that proprietary to us. Actually one of our customers reached out to us. It's a fortune company that reached out to us and wanted to run the same program us.

15:55 We didn't design the program to run it for other people but you know it's a customer enterprise customer and a big customer. So we just wanted to make sure that we are able to support them as well. We ran the program for the customer and the outcomes were very very similar. The customers are also seeing acceleration in the work on their side. And again, I know time is up, just one second. I think if any of you attended Andrew Ing's fireside ch chat yesterday, one of the things he was talking about, he's worried about lack of education or lack of investment in education by a lot of people building AI because that is going to become one of the barriers to adoption. And so we are sort of seeing I'm giving you a data point from our experience in support of that claim. And we are saying something very similar that you know education is an important component of getting AI deployed in enterprises. just a little bit about ironclad very quickly.

16:52 as I said we are almost all Silicon Valley companies are customers for us including all the frontier labs. The ones we have logo we I'm putting the names down at the bottom have thousands of customers and so on. that's it from my side. I am going to be around in networking sessions or breaks if you want to chat with me and this is my LinkedIn in case you want to follow. Thank you very much for your attention.