Why AI Companies Are Buying Your Data

0:00 This founder is running a $4 billion company and he's only 25 years old. Ali, the CEO of Micro One, is building one of the biggest players in the everp popular hyperrowth data labeling space. On this episode, he gives me a masterass into how data labeling works as a business, how the industry could go from 30 billion to $1 trillion a year, and why building in this space isn't as easy as a lot of people on X may think. Let's hop into it.

0:30 What first attracted you to the business? Because I think similar to the other businesses in the space, you didn't start in this space. So, what created the initial intrigue to completely pivot the business? >> I had a company when I was in undergrad at Berkeley called Moonche. It was a software development agency. It was nothing nothing special. It was just like serviceoriented company. Built built websites and apps for for other companies. And one of the kind of you know limiting factors at that company was every time projects would come in I had to interview a bunch of engineers and assign them to the projects and kind of scale up and down the the teams. And so a lot of my time was spent just interviewing candidates. And I ended up building a tool that essentially helped me with with with that bottleneck. which was this screener, this AI screener that kind of helped vet some of these engineers and give me a report on how they did and then I would sort of talk to the ones that that did well. That AI screener became the first version of Micro One and it was the sort of like it was the first ever AI recruiter in the market. and at first there was u you know nobody would accept an AI recruiter. There was like a ton of hate on on Twitter about it and all all that. But but but but very quickly it became you know kind of normal to to have the a vetting process that is AI oriented and and so the version one of our business was was that which is like we allowed companies to use our tool to source some vet talent which was engineering talent but also like other types of talent. And then we also ended up building kind of like a next generation marketplace you could think of it as which had a bunch of prevetted talent that had gone through this process which companies would be able to hire from. And this was we worked on this for like you know like a year or two and and and kind of scaled up to a few million in run rates and then we we realized that this is you know there's a sort of data industry that is evolving quite a bit to the world of experts where the data that is that is needed to continue pushing the models is expert data. And naturally one of the things that you need to solve for before you can kind of create these data pipelines is you have to recruit those experts. So we we had this realization long time ago and we sort of decided that the best application of this AI recruiter that we built is is going to be the the data market and we just kind of went went all in.

2:49 >> You know obviously it's orders of magnitude bigger today than it was two years ago even if you just think about the spend that the labs and other companies are putting into the space. what what dynamics cause that to happen? It's really three things which you know is sort of commonly said just compute data and algorithms and algorithms is one where there's there's enough you know there's sort of a lot of iterations and innovation happening and a lot of the researchers at labs they sort of focus on overall architectural innovation and and algorithm innovation and then compute of course there's like tons of people working on this there's lots of lots of investments lots of data centers being built out and it's still very scarce but but I think over time it'll sort of become become less scarce the thing that really pushes the intelligence of the models forward is is the data. And that data is you know the focus of of of all all the data companies currently and and essentially like if you look at the pre-training phase there was the internet which was a massive data set that humanity has created for decades and so models had a huge head start from this like really big SF essentially SFT data set existed and there was a lot of like progress made because of that but then once that was used up. The way you push the frontier in different areas is you need you need human experts that that sort of distill their judgment into what's called RL environments to continue pushing the frontier. And so that that's that that's why data spend is going up rapidly is because there isn't open largecale data that exists beyond the internet. And so now all the post- training that's happening requires kind of net new data creation which is expensive but but there's lots of ROI that comes as a result which is why spend you know continues to go up and and our our bet is that the spend is actually we're sort of in the very early exponential of the spend increasing and there's sort of a couple of orders of magnitude left in increase.

4:47 >> What does it look like to operate a business in this space? What is required? What is the endto-end process of a contract look like? like paint a picture of what operating a business in the data space looks like. >> We are delivering what we call units of intelligence improvement. This is like this is the abstract way to think about it is the customer wants to buy from us their system improving in its intelligence whether it's a model agent or or anything else. And so it's it's our job to do that in the most efficient way that we can. And there's a lot of parts to to make that happen. And I'll I'll walk you through a few sort of examples. The the first is whenever there is any given pipeline that we're setting up to improve any, you know, model or agent, we need usually hundreds of of of experts or sometimes even thousands of experts. And those experts need to their skill sets need to match for whatever improvements we're actually looking for in a very specific way. And it's not, you know, if we're trying to improve a model in legal, let's say, it's it's never as broad as like, okay, let's just hire lawyers. It's like let's go and find M&A lawyers that have this specific specialty that are very good at writing legal memos for example.

5:57 And so so so our our vetting environment is set up specifically for that and and and and does a very large scale set of interviews to sort of make that happen. That's the first that's that's sort of the first step. Just one question on that step is the way that like this actually happens just playing out the example you were talking about is one of your customers whether it's one of the labs a large enterprise is that they come to you and they say hey this is a blind spot in our model in terms of performance let's just use the example of answering queries for M M&A lawyers at cor corporate law firms or helping them redline and then you you say okay and then basically like how gran basically how granular do you get to to or do you have to get to to know basically what are the experts you need to source and what are the effectively like the questions or environments you need to create for them to get the output you're looking for >> there's two methods one is either the lab knows you know what is the pipeline design they know exactly where they want to improve they know the data tonomy and the QC process etc and they sort of give us a spec and then we go and set up the pipeline or we find failure modes in in in models. So we we go to you know lab and say look your model's failing in this this sort of M&A memos or or or this this area of medical whatever it is and here's the pipeline that we suggest and and we sort of already designed it we already have some tasks set up for it you know and do you want to scale this up so there's a huge research component that is increasingly important where we are kind of diagnosing model failures and then going and solving that whether or not it's it's the lab designing the spec or or us there's u there's the process of first vetting large scale of experts. but then the the important part is is is really like turning the human expertise, the raw human intelligence and expertise into training signal and and and that's like the core of our product is that which is of course like the the vetting of of skills and getting the actual human is still a very important part but the core of the product is taking all that and actually turning it into environments, verifiers and just overall training signals so that we can train models. So I'll give you a couple of like u you know components of that. The first is our data platform which is basically where the experts do the work. So it's like where they create prompts that models are failing in. It's where they create the rubrics and verifiers that describe what a good outcome would be. It's also where we have our data generation models which are essentially helping the experts do their work. So a data generation model would be you know helping an expert kind of create a verifier or or reviewing a verifier that an expert creates. and or it might be like you know helping transform some some PII that exists within like you know the seeding data of an environment.

8:43 There's there's a bunch of you know different examples of what data generation models do within our our data platform that allows the raw human expertise to turn into training signal. And then the third component you know broadly is we have to very closely manage the performance of all these pipelines. So you know and what it means to measure performance is what's the data quality like and and we have to have a very specific kind of definition of of data quality for each pipeline.

9:07 we have to see what the throughput is like and and and and there's a bunch of KPIs like that that we have to sort of look at very closely to ensure that the pipeline is scaling up correctly. So all these things come come together that allows us to you know ultimately improve model capabilities. And I think there's a there's sort of a misconception in the market where like you you could just take like some arbitrary data or like raw data and just and and and and just sell it. There's a little bit of that right now just because there's a lot of there's an emphasis on company operational data which we which we can get into. But but the real value from like even company operational data is is the downstream training signals from that downstream environments that are created from that which is which is the the real focus of of what we do. Is it quality data and is it actually driving improvement for the models? Is that is that what happens in the third step?

9:55 >> Yeah, exactly. Basically the way to think about third step is like we want to we want to drive what we call predictable and abundant intelligence improvement for our customers and predictable is what you said which is like the customer needs to know is it actually improving the model and and that like you know that there's like a very very kind of specific way to define what you actually wanted to improve on what benchmarks you care about what what what you want to make sure it doesn't regress on and so forth.

10:22 So we have we have like a careful tracking of that. And the other one is we want to get the cost to to continue to to to be lower over time, which is the sort of abundant part. And so if we can get the data cost of useful data to kind of approach zero quote unquote over time, and it's infinitely predictable, then it's then it becomes very very high value. >> You talked about how either like there's two basically two ways that you decide which experts to go after and which tasks you want the experts to do. It's either a customer comes to you where they know a blind spot in a model, they provide a spec on what they need or you're testing the models, you're finding blind spots and then you are creating the spec yourself r like in terms of buckets and whatever you can share what actually is in the spec before you actually go into the process of finding experts.

11:08 >> Let me let me describe what what we call failure mode reports look like. >> So this isn't exactly a failure mode report. This is a this is a benchmark, but but but they're they're not like too different. Like when you when you look at a benchmark, you what what we're pointing out mainly within it is like what are the things that models are failing on? And let's just talk about personal agents because that's like you know everybody's talking about them right now. So I think it's a relevant topic. so we released recently this u this personal agent bench which is essentially a benchmark that tests how all these different personal agents do in everyday tasks. And it it tests the sort of ability of real delegation without doing things that a user wouldn't want which is like oversharing with certain you know platforms that it's booking something with or like booking a flight without asking or you know like finding information that is not necessarily relevant or too verbose. Like it basically tests for all these things that users actually care about. And it looks at two main things which is trusted completion as well as the just the real kind of completion that happens. So when we talk about a report like this like when we release a benchmark like this the the key thing that it focuses on is where are the models failing and what are some of the areas that can be improved and then we sort of score all the all all the models on on the overall kind of success rate as a result. And th this is this is what results in areas of improvement which is what results in the specs that I was describing earlier. And I'll just show you a quick like funny example. You know Brett Adcock from from Harkc you know he sort of like quote repost said he was like how embarrassing. We're going to work hard to crack number one because they're they're number two relative to to dots.

12:58 So, so, so basically, this is like this like failure modebenchmark, which are not exactly the same, but like they're they're sort of similar, results in, labs coming up with pipelines to then improve in areas that the benchmark was describing as as failure. So, what we do is when we publish these publicly or we have like private reports that we create, which we send to our customers, those result in a pipeline design spec, which includes usually the domain that you're focusing on. So if you're let's say you want to improve in this like personal agent category, you would focus on there's like many permutations of of pipelines you could set up. One could be you focus on like the this the privacy and and safety. So like you don't want u you don't want the agent to to overshare. So you would probably focus one pipeline on on just that fact of like is the agent overdoing the information that it shares with with with counterparties that it's interacting with. and what's included in that is again like the the subdomain you're focused on but also the data tonomy which is you know do do we want to go for a rubric based design do we want to go for preference labeling you know sort of arena rank style and you basically decide like what is the data type and and overall tonomy and then from there you you decide what is the scale of data that you need. like you know and and the scale of data is usually based on what what you think the downstream model kind of RL run is going to require. It's usually like hundreds of data points or maybe thousands sometimes tens of thousands and and then you you know based on that scale you say okay we want to get this done in a month two months three months and then you ramp up the number of people required plus the data generation models and computes required to actually create those tasks. and then and then you and then you set up the pipeline, you create the task and then you sort of go go from there.

14:44 >> What are kind of the different buckets of data taxonomy and how is how is it decided in kind of the specification process which taxonomy makes the most sense for the specific kind of capability you're trying to increase in performance in. >> Let me explain two categories and I'll go deep into them and and I think this will sort of convey the point but you could think of it as there's like many other many other categories as well. So the first is what what we would call kind of evaluations broadly which is and it's you know bit of a bit of a broad word but usually what it means is you are having experts ultimately evaluate the output of the model and you're doing so in most cases through some sort of what's called liyker rating which is basically the the human expert is answering a bunch of different questions about the output and it's sort of they're they're rating the performance across the dimensions that you've defined. So you know if if we take the the personal agent example one of the questions might be you know did did this task what was this task completed in a robust way in the way that you kind of expected rated one to five five is like 100% completed one is like you know it failed pretty much completely the other question could be was the was the output that it gave you you know like brief enough or or too verb verbose, right? And then you would like create it as like way too verbose or or like you know, perfectly enough information and and not not anything extra. and then you know, so so you could think of it as like the eval u kind of liker scale rating tonomy would be humans just rating the output with like sort of like almost like a survey. And this structure of data is very useful to get u to get sort of a final sense of how the product is doing usually before release. But but this data is not in most cases is not reproducible when you when you when you come up with a new model because because you have to then rerun you have to have you have to bring those you know experts back and say hey can you actually just answer the questions again all you know for for for this like slight change that we created. can you answer all these questions again and and like run through thousands of you know tasks again. It gives you a lot of signal and it's very useful for like very subjective you know questions and so forth but it's not super scalable because you can't rerun and reproduce with new models that come out. Now another structure which is like the structure that that's used very very much so for for like actual training runs and and RL runs is is this rubrics texonomy which is you get experts to instead of answering these questions and like rating it and so forth you get them to actually describe with a bunch of different what's called verifiers what a good outcome would be. So, so let's go to the legal memo example because personal agent is it's harder to do rubrics for now just because there isn't like too many things that define a good outcome. Like it's lower dimensional, but if you talk about like a legal memo, there's like many attributes you could point at that that define what a good quality memo is. So for a legal memo, you would, you know, you would look at the, let's let's say you're like working on some sort of acquisition of a company or something and and maybe you know, maybe the legal memos on on on like whether the acquisition is recommended and what the price should be and and you know, something like that, let's just say. So what a good outcome would be is like some valuation range that you would expect the model to give. so you would say like valuation range must be in X to Y. you know, another another thing would be like maybe you're maybe you're expecting the the acquisition recommendation to be like a no or like or like close to a no. So you would you you would say that and maybe there's like a bunch of qualities of writing that you care about based on what your enterprise sees as like a good memo. So, so essentially you would create like you know 20 30 maybe 40 rubric items and and and a legal expert would create this maybe in collaboration with a finance expert and and and these are what's called binary rubric items which you can have a model grade another model with because you're sort of very objectively saying this this this thing should be included and this other thing should be included and no matter like what the model's knowledge is in and legal memo or legal broadly it can just check if that thing is there and that's And this is and this is just LLM as a judge.

19:10 >> Yeah. Yeah. It's called a great greater model. which is like similar to the LM as a judge concept. So so so the point here is that this is >> it takes a lot more effort per task because you're not answering the questions like yeah you know looks good whatever you're like defining bunch of items and you have to be very precise about them and you have to make sure that it's representative and and comprehensive of everything you actually care about. But the beautiful part is that you can rerun this which is one you know let's say you have some model model X and then you you you build model X plus one you can then say okay now how does it do and and there's the human experts don't have to re look at it again and rewrite anything they already wrote the objective criteria so a greater model comes and says okay now that model X plus one gave me this other output let me let me rerun it on these rubrics and see how model X plus1 did and there's a score because each of the rubric items gives do couple points if you do well on it and then if you get the score to be high then then then then great and and you just sort of keep keep repeating that process.

20:13 >> What are from your perspective the hardest things operationally to make all of this work as a business? My guess of what's really hard here is the type of scale you're working at getting this many experts in a variety of different domains. My assumption is pretty quickly because I'm assuming like the the the lead time that you're given to gather experts is not super long. I could imagine that's really hard. I could also imagine setting up the right environments is like operationally difficult. And then I can imagine like the data quality piece could be difficult. Take me through one are these the hard pieces and is there anything I'm missing?

20:55 >> There's a massive scale of experts and that massive scale of experts has two core really difficult components. The first is vetting them in a deep way which you know as we talked about we do through sort of AI process and if you you know if you imagine doing doing this manually it's like we would need we we we interview roughly like tens of thousands of people a day right now and there's over 10 million people that have kind of go gone through the process and if we were to do this manually it would be like thousands of recruiters and like you know subject matter experts and so forth. So it's like not like not feasible. So you have to build a really good endto-end product for that. The second hard component of of of man of experts is there's a there's a very large scale like managing of experts required which has you know what we call SPLs which are strategic project leads that are kind of running the pipelines making sure the experts are understanding what's going on you know doing office hours etc etc. So so so there's that component. The second component as you mentioned is quality of of data is is a really key component and it's a it's it's it's one that's very difficult to get right and there's many aspects of this but the first aspect is like you need to ensure that the experts that are defining the the objective criteria and rubrics for example are actually like doing it correct. because a lot of a lot of a lot of subjects that we work in or domains that we work in are quite are quite subjective in nature and and law is even like a pretty good example. I would say like most domains are actually quite subjective and so if you look at if you look at you know law for instance you need to have a lot of peer reviews and so you have to build a system for peer reviews which is like lawyers that review each other's rubric items and like send it send it back and like you know leave comments and so forth and then you also need data generation models to come in and also do AI based quality assurance. and then also in in in quality there's the increasingly a component of models trying to hack the like reward hack and and basically like try to find the answers. And actually let me show you a quick quick visual here.

22:58 >> Yeah. Yesterday we had this thing where we essentially observed Gemini trying to hack its way through the right answer and and this this happens this happens quite quite a bit and and and like humans can't always catch these and a lot of cases they they they can't catch it at all. So there needs to be a lot of systems in place that are kind of AI based to to catch these. And so like in this case basically Gemini 3.8 Flash was searching the sandbox for a bunch of hidden it was searching the sandbox for hidden files that actually included some of the answers and our flow QC system basically detected this and and sort of stopped it and other models like GP6.1 Astra cloud etc weren't doing this and so this shows like you know we're building a lot of technology around ensuring that the scores that the models end up getting as they do these rollouts are actually correct. And I because because if you if you give it the wrong reward with with the model just like finding the reward through hacking, what happens is that models they will do more of that and and they just will increasingly become misaligned.

24:11 because they they will understand that they can easily optimize for cheating and if they do they will get higher rewards and they know that they're being evaluated. and so they really try to get high rewards which which like if if you if we sort of don't stop these quality issues this poses a pretty fundamental risk u because the models just end up becoming like misaligned and and and and like they they sort of start thinking as like cheating is is sort of a is sort of a good thing in these evals. so so a lot of these like quality checks are like a huge engineering feat for us to for for us to ensure that that models are sort of doing a good job. I'm an M&A lawyer and I've been brought in as an expert by Micro One to effectively create rubrics.

24:57 And so the M&A lawyer who's providing the rubric would basically provide a list of criteria around what they would expect the model to to answer back with in relation to the question I just asked. Is the M&A lawyer actually providing the rubric in the way that they need to provide it? which is why you have all these peer reviewers. You also have like kind of technology helping to support in this. but then you also have where if the model somehow gets access to the rubrics that's been used, then it's just going to provide an answer that appeals to the rubric without actually being trained on the right way to behave.

25:38 >> And and those are two examples of how quality can go wrong. But there's there's there's like many many examples of how quality can go wrong. And so and the ways to solve them is it largely comes down to two things which is you need to keep building better and better kind of peerreview systems and then you need to keep building AI based quality assurance systems that can catch all all these sort of different things. but yeah those are kind of the two main categories.

26:04 What do you think this space will look like say 12 months from now? And feel free to change the timeline if you think that's the wrong period of time to look at. this space is in the ver very early beginnings of the sort of exponential growth that we're going to see. I think the sort of number we have in mind is in the next two to four years it's very likely that the space will be a trillion dollars a year kind of overall spend space and and if you look at >> what is it right now >> yeah if you look at what is it right now it's like it's roughly I would say anywhere from like 30 to 50 billion and you know growing very very quickly but but roughly in that range and that's including you know labs robotics as well as enterprises and it's of course largely the labs there's simply no other way to keep improving models other than than than data and and of course that data can be u human generated and synthetically generated and the percentages change and and over time the percentage will will be in fact most mostly synthetic which we'll touch on in a second but but at but you just need you need data in in in one form or another to to keep pushing the frontier to keep pushing the models forward I think the if you look at all the compute bets that are happening you know which are sort of u compute bets that are that are sort of betting on the fact that inference is going to improve increase by huge amounts over the next decade. Th those assumptions are based on a lot of new model capabilities being unlocked and and and if if those model capabilities are if we don't sort of continue unlocking lots of new model capabilities then then then that'll be sort of a disaster for for all the compute investments that are happening.

27:45 So there's sort of like no choice but to keep improving the intelligence of the models and and and and therefore there's no choice but to keep investing massively and increasing amount in in data. and so so that that's sort of the first thing. The the second thing is there's there's this notion that that like what we are distilling into the models, you know, the human knowledge that we're distilling into the models is like this like static knowledge base that that like once you distill, you know, once you distill it, you're sort of done and and and you know, the models are are now know all all of humanity's knowledge and we're good to go. But that's that's that's like the wrong way to think about it. Knowledge and and overall like knowhow is just like completely dynamic. And the example that I always like to use is if you think about I think there's some sort of stat where it's like when when a medical school student graduates, it's like something along the lines of 30 to 40% of their knowledge is is either like irrelevant or they have to like relearn, you know, they have to update it in some way. And that's like a few years, right?

28:45 like in a few years a large portion of of of some important knowledge that you've learned is no longer super valid or or you have to like you know update in in some in some form which is an example of like how dynamic knowledge is and and of course you can you could take this argument and apply it to law which is like probably even more so dynamic because like we're introducing new laws and it's like very subjective etc. It's also like interestingly self-fulfilling, right? Because it's like as the models get better, >> the the speed by which the >> the tribal knowledge you need to be successful in your job arguably changes by at a higher rate than it's ever changed.

29:21 >> Exactly. Yeah, >> that's that that's exactly the the second part of the argument which is like knowledge is dynamic and what pushes knowledge forward is is at least partly the the tools that we use and and and the the inventions that we have that that results in us kind of figuring out net new things but also changing the way that that we do these sort of exist existing things. This is like a positive feedback loop that gets created which results in this notion of what we call kind of infinite last mile which is you don't actually have a last mile and and and and the last mile actually accelerates and it sort of expands and and so the there is no there's no ending to like aligning models with humans because humans are just updating in the in the way that they think and the way that they they do things and so forth. So and and I think more practically there's like if you think about enterprises and robotics there isn't much data spend there yet. I mean it's it's very low and and and even if you think about the labs the areas that they're mainly focused on is really just like four domains. Medicine, finance, legal, and coding are kind of like where most of the data spend is in.

30:28 And there's many more domains to to go after. It's just that there isn't like we need more data efficiency and and lower cost of useful data for for those domains to unlock as well. So which brings me to the other point which is as we as as we sort of make intelligence improvement more abundant through data generation models and through synthetic data the amount of new domains that will be unlocked and the amount of enterprises that will be able to kind of spend in this area will will will exponentially grow. so I think we're we're in the very early days of of data spend and I think in the next couple of years like two to four years it'll be comparable to compute spend.

31:02 and there will be you know kind of probably three to four companies that kind of take take large portions of that. One argument that I hear often is like people trying to wrap their head around if they agree with the notion that enterprise is going to be as big of a buyer of data or like of kind of what you sell as the labs. Explain how you think through this and why your bet is the answer is yes.

31:29 >> It's not going to be the same as the labs. >> Yeah, it it will be onetenth the labs on average. But there's but there's you know thousandx more enterprises than than the labs. The the the ballpark is labs you know neolabs and and big labs and sort of all labs are focused on mainly foundational model building. Their ballpark kind of spend will be in the billions like maybe even tens of billions but but let's just say like rough roughly a billion. Our bet is that enterprises like Fortune 500, Fortune 1000 broadly will converge to an average of about 100 million a year in spent.

32:04 And it sounds crazy and you know obviously it's going to be like a huge part of their their their P&L but but but the the reason why we say this is because this this budget from enterprises is not going to be it's not like a software budget. It's not like SAS or it looks a lot more of like core IP almost like payroll budget where so so so if you start to compare it to like payroll it's like oh okay it's like you know 1% of their payroll five% it becomes like a very small amount actually and if you if you think about like you know what is the sort of most common theme right now within enterprise it's it's like how do we own own our own intelligence right to us owning your own intelligence means that you're defining what it is that you care like what is the intelligence that you want to own right You can't be broad about it. You have to be very specific on areas that you want your enterprise to have sort of intelligent systems in which is like agents in. And then you need to see if you are actually getting the agents to be intelligent in those areas that you want them to be intelligent which is like you have to measure their intelligence which which is which is eval and and and as you build this this very sophisticated what we call evaluation stack and and you know kind of measurements of of of Asian intelligence you're building very core IP which is and and it and it doesn't it doesn't matter if you're using like closed source models or open source. I think that argument is like kind of irrelevant. I mean, of course, there's like data retention and so forth that matters, but but but if you but if you if you look at the the one of the key things there is like are the models swappable or not, the answer is they absolutely are if you have very carefully defined what you care about and you're very carefully measuring it through evaluation stack. So, so our belief is that the the most important component of of of enterprises building their own intelligence is having a large amount of investments in in eval and orchestrating models models properly and and that doesn't necessarily always mean that you have to post train like those are not equivalent of like if you're doing evals you're only using data for post training that's not necessarily the the same thing. so yeah, that that's sort of how we think about >> that makes sense. Just to make sure I understand this, it seems like the enterprise buy like what they are buying from you all is like orthogonal to what the labs buy and I want to make sure I understand or don't understand which is like my understanding is the labs are buying these large data sets from you all to do RL on their models for enterprises. My understanding is like they already have the data. Now the question is is are they accessing the right data? How do they organize it? But what you're talking about is really like the eval infrastructure like the the foundation of the home to be able to benchmark how whether it's closed source models or whether it's fine-tuning models that again it's irrelevant but it's like are they able to evaluate how models do or agents do on tasks so that they can easily basically hill climb to wherever whatever they want to achieve. Is the buy by an enterprise different in the way I'm understanding it?

35:07 >> Everyone talks about this notion of like enterprises have a ton of data that they can use, but it but it's always this like it's abstract like how do you actually what do you mean by that? Like how does an enterprise use their data? Right? >> The way so there's two things there. The first thing is that in a lot of cases they won't and can't actually use their enterprise data because a lot of times it's like customer data and you know you can't you can't train on customer data and and and you can't even evaluate your product on customer data.

35:36 sometimes there's internal data that might be useful but that's not like you know that's not necessarily like what what people point to when they say enterprises can use their own data. So first of all we need to like define what that means and are enterprises actually doing that. The answer is like no not really. The second thing is that the way the way you sort of use your own data which is not necessarily the most like important part of this whole thing is you you have to build environments which you seed with realistic data which might be your data but in a lot of cases doesn't have to be like we have some enterprise customers where you know they want to they want to evaluate how their agent is doing on a representative customer use cases and of course they can't use customer prompts and outposts and so forth. So they need to build environments that that represents their customers. So in that case, they're not even using their their own data like pretty much at all. They're they're just like telling us what is their customer distribution roughly and then we go and build those environments that match that distribution which is like we have to go and find companies that that that have operational data that like roughly match the their distribution of companies which is difficult because the distribution for these enterprises is usually also on other enterprises. So the operational data we get is not necessarily enterprise. Usually it's like usually SMBs. So we have to like extrapolate data, you know, make it make it more make it more look like more of an enterprise. but regardless with it, we take that data, we seed the environments with very noisy operational data and then we build verifiers on on what the enterprise cares about. So a lot of times the environments that get built are representative of one's internal data but not necessarily like actually using their internal data. But but but the important part is like you have to know I I think like one way to like think about what it means to like use your own data is like what is what is it that you actually care about within that data and then you like replicate that within environments that that's kind of like a better >> super interesting one way I think about this is a big reason the labs need a micro one or other players in the space is twofold one is like the process of get the operational like exhaustion of getting all this I assume they believe is like not core for to their business, but it is necessary for them to get kind of the the end outcome for their business. and and also like it relies on this network of experts. It sounded like you were alluding to that more and more of the the balance is going to move to synthetic. Then the question might one I'd be curious why you believe that's the case and why you think synthetic data like one thing I've always heard is like as you get more and more synthetic data there's a question of how does RLing like do you see see the same improvements in models or not but I'd be curious as you get to more and more synthetic data why does the incentive not swing for for labs to just produce that synthetic data internally once they have bought enough data now I guess the argument would be data is dynamic so you're always going to still need net new data to create synthetic data off of >> the core one of the core products that Microwana is building is data generation models and and let me let me explain to you like what this means exactly. If you think about a pipeline like you know just imagine the data platform that we have expert goes in there creates tasks as we were discussing earlier for a legal memo. So where where where is synthetic data come in? One example would be as the lawyer is creating these these rubrics.

39:05 They actually don't create the rubrics. They create the memo itself which would be much faster than creating the rubrics because they would have to create the memo and then the rubrics. So one example would be that the the lawyer spends their time creating a very good artifact which is the memo itself. And then we have we have our flow gen model that that we call it. use that artifact and generate the rubrics on its own where it it it you know looks at a high quality PDF and it says these are the components that I think define this high quality PDF. So in a way it has to depend you know it has to depend on its legal knowledge you know a little bit but it's more so depending on its of its kind of extraction of training signals from from what's already grounded in in in in human knowledge which is that that expert that created that middle. So in that case and and by the way like right now synthetic data is not at a place where like you can do that but but let's imagine we we get there where it's like you take a memo and you can just automatically create rubrics you know all day long. At that point you've cut the expert time by probably 50 60%.

40:12 You've made the pipeline way less costly per per data point and you've made it more more efficient. But but you still need that expert grounding and you still need that expert network. you you still need to to scale up the experts. In fact, you need to do it probably more now because like they're really useful. So like let's actually like double the data that we're getting because it's also cheaper and maybe triple it. so so what happens is and and you can just extrapolate this very much so which is like we get to a point where the model does 99% of the work like you geometrically approach 100%. You never hit it but you get you get very close.

40:43 And so like at that point the model's like creating the artifact, creates the rubrics, does everything. the experts just kind of saying, "Okay, like this looks good. Signing off, you know, we we go from like 15 hours of of of of of creation for that one one task to like five minutes maybe." and at that point, environments and data sets are so are are so low cost and so abundant that we're going to scale them up like crazy.

41:12 Like the amount of pipelines are is going to be, you know, thousandx from what it is now. and every company in the world is going to want environments. Even though experts are now like 0.1% of the data generated in the pipelines, we now need way more experts because of how many pipelines are being kicked off. And it's like typical like Jevans paradox and we're seeing that play out and and and we're seeing it playing out in coding which is in coding it's it's already it's is largely synthetic because models are very good like you know it's like 70 80% approximately models can kind of generate rubrics and and and many other kind of data types.

41:47 We need more coders than ever right now like we're literally like increasing the rates by a lot. We're just like we can't get enough and and and so this it's it's happened already for an area where like models are super human. So one way to think about this more broadly is one architecture of RSI is like basically continuously increasing the percentage of of of data being generated by models for any given pipeline and if you keep approaching 100% which I think you don't hit it but you but you get close that's like one I would say the the the main architecture of our side that comes to mind and the value is largely acrewing in data companies because data companies are are the ones that for again orchestrating that that whole thing. Now the second thing that is really important in this is we because we work with all labs and and you know bunch of enterprises and and robotics labs and and have like a very diverse set of customers. We we get to have a very large and diverse a very large volume and diverse set of pipelines which and and we get to have experts that like refine whatever our data generation models are are creating which builds a natural feedback loop for the models that we're creating for data generation and and it creates this natural feedback loop in a very diverse setting which like any individual lab doesn't have that because they only have their own pipelines. so so like this the scale can be much much larger specifically for for data generation and that's why we're building like so aggressively towards this because if we like I if we get pipelines to be more efficient through data generation it is like the best thing that can happen for our business. It's not like a it's not a bare case. It's actually like the most bullish case - which is this idea of RSI >> and and so just to play basically play this back to you the idea is that right now lawyer is creating the artifact and then also creating the rubric. Imagine a world in which the model starts doing a little bit more. So the lawyer creates the artifact, the model creates the rubric, the lawyer just checks the rubric. Now instead of spending an hour on the task, the the lawyer spending 30 minutes on the task. You take this to a natural extreme where ultimately the model is creating the artifact, the model's creating the rubric, and the lawyer's just checking and maybe they're spending five minutes on the task. As that happens, there just ends up being basically more and more abundance of demand for data. The the view is like there's just kind of this in infinite dem like almost infinite demand for data. And what happens my assumption from a pricing perspective is even if let's say the labs on a per task or whatever the data point is or the the unit is even if the labs are willing to spend less per task or less per minute you're you see margin expansion because more and more is being driven by synthetic versus the humans a and so then it really becomes margin expansion on maybe a lower dollar per task value but the number of overall tasks completely balloons.

44:56 >> That's exactly right. Yeah, you summarized it very well. The only thing I would add is there's a yeah, the way marginal expansion happens is the percentage of of effort and of of like data being generated by compute is now almost the whole thing, right? and and and the margins is on is is on computes which is very controllable versus like margin on on on expert hours. A and also like if you imagine a world where intelligence improvement is abundant like you can just it's free.

45:28 It's almost free to improve your intelligence systems and it's predictable which is like you know exactly how much you're going to improve based on what you're paying which are of course they're both like theoretical you know concepts so we we we can't we can't get to them but like we can strive towards both of them. So, so the statement we like to say is like if you can buy IQ points for your systems and it's free to do so, you will buy all the IQ points you can get and and so like that's like the infinite demand sort of exponential that we'll that will strive towards.

45:58 >> Last question for you before we hop. This is the the white elephant, which is you've basically like you you and the the micro one mafia are just like all over Twitter Twitter buying every SMB's data on planet Earth. Just like just give me the Yeah, give me the story around like why now? Like why is this happening now? What's the thesis around this? >> If you think about what does an RL environment include, there is there's the seating data, there's the tasks, which we talked a lot about, and then there's the tooling. the tooling is not not super important but you can think of tooling as like basically the models and agents they need to use some sort of software to do their job in any given environment. So so that's part of it as well. tasks we we spoke about but the third part is the seating data which is like the world that the model has access to as it and and it like refers to as it does a bunch of tasks that you care about. And if you can if you can make this world very realistic, you you're you're you're training the model in in in a in a sort of an area where it matches what what it's going to do once it goes to actually work for any given customer, once it goes and does does real work within an enterprise. So there's this idea of what we call kind of scaling on realism. It's almost like a sort of a new scaling law like quoteunquote. There is tremendous value on on scaling on realism on downstream model improvements and and the way the way you do so one of the main ways that you do so is you you purchase company operational data and you you know you license that and you anonymize it and you have to anonymize it in a way where you have to like transform the data versus redacting you have to transform the PII versus redacting so you can keep the noise intact you can keep identities intact and so forth and then you seed a bunch of different permutations of of environments to to then to then you know create pipelines just just like the the way that we do normally. So yeah, basically the summary of it is like you can you can scale on on how realistic the environment is as a whole. And the way you do so is you scale on each of the ingredients which is you make tasks more realistic which you know we've been doing. You make the tooling more realistic which is you get the the software that's the simulated software that's used to be more and more real like if you're if you're using some like docuign type software you make it as close as you can get to to be like docuign >> and then you make the seating data more realistic. And and the third one is what the company operational data is is sort of about. So, let's just say I'm one of the biggest law firms in the world, and I'm trying to build an agent that helps, we'll use the M&A example again, an M&A, an agent that helps an M&A lawyer with with redlinining on contracts, the same type of contracts. Maybe it's like LOIs that we see over and over and over that enterprise if they want to build this agent that helps with that task for whatever reason for for a number of reasons they may not want to train on their data. let's say it's because they don't want to like they have all this PII customer information whatever would the idea be that you are buying a local law firm's data where there is basically operational data and I'd be curious what that means in context around how a local lawyer is doing contract redlinining for maybe M&A of a small business in the area and then that is basically being extrapolated to what the environment would look like for this enterprise without using their data.

49:22 >> Yeah. So for for for enterprises that that's the that's the right way to think about it and it's very similar for labs, you know, sort of similar concept. >> law firm is is not the not not the best example because we don't we don't work with law firms because it's like bunch of privilege data. But but but you could think of it as like just a SMB that h you know that that has a bunch of >> desk workers that use Slack, use Google Docs, use different tools and so forth.

49:49 We take that data, we anonymize it using our PII transformation model and then we seed environments to to help train models. So for enterprises, it's exactly what you said, which is like a massive enterprise maybe wants to test their agent on how their customers are going to use it. If they want to test their agent on how their customers are going to use it, they need to replicate kind of a digital twin of what their customer is going to look like as they use the tool. And the way to do so is is is to create this representative seating data from from company operational data.

50:19 >> Awesome, man. Well, I think you gave me the the full download in the space. Really appreciate the time and I think this is going to be really helpful for people as as people take to Twitter to learn more about what's going on in the space. >> Yeah, absolutely. Thanks, Alex. Thanks for having >> Thanks, man. Appreciate it.