transcribe

The Jevolution, Explained: What Jev Actually Is and Why Everyone Is Trying to Copy It

The MAD Podcast with Matt Turck · 1h 7m · transcribed Oct 2026
More from The MAD Podcast with Matt Turck Business
𝕏 Share ▶ YouTube 📥 PDF 🤖 .md

Section Insights

# 0:00

Introduction to Jev and Its Impact

What is Jev and why is it significant?

Jev is a new AI model that has quickly gained attention in the AI community, becoming the most copied model shortly after its launch. Its significance lies in its innovative approach to AI, which some insiders find difficult to explain, while others underestimate its potential.

  • Jev has rapidly gained popularity in the AI sector.
  • It represents a new kind of model that challenges existing perceptions.
  • The model's complexity and potential are not fully understood even by experts.
# 13:35

The Future of AI and Composability

How will AI evolve in terms of workflow automation?

AI will become more valuable when workflows are encapsulated into reliable, composable units. This approach allows for the creation of higher-level functions that can be reused, leading to more powerful AI applications.

  • Composability is key to the future of AI automation.
  • Reliable building blocks will enhance the capabilities of AI.
  • The evolution of AI will depend on creating modular and reusable components.
# 27:11

Understanding Reinforcement Learning from Human Feedback (RLHF)

What is RLHF and its role in AI development?

RLHF, or reinforcement learning from human feedback, is a method that optimizes AI models to interact more effectively with humans. It focuses on making models conversational and stylistically pleasing, although it has limitations that necessitate the development of new AI families.

  • RLHF enhances AI's conversational abilities.
  • It has stylistic optimizations that may not improve factual accuracy.
  • New AI models are needed to overcome the limitations of RLHF.
# 40:47

The Importance of In-House Data Creation

Why is in-house data creation crucial for AI development?

Creating data in-house is essential for developing effective AI models. It allows for tailored iterations and a deeper understanding of the tasks at hand, ensuring that the models are built on a solid foundation of relevant data.

  • In-house data creation is vital for effective AI training.
  • Understanding the task is crucial for developing reliable AI models.
  • The current landscape lacks clarity, making in-house expertise essential.
# 54:23

Critique of Current AI Benchmarking Practices

What are the limitations of AI benchmarking?

Current benchmarking practices in AI can mislead users and do not accurately reflect the reliability of models. The focus on benchmarks can detract from the actual performance and usability of AI systems, leading to a misunderstanding of their capabilities.

  • Benchmarks can mislead users about AI performance.
  • Optimizing for benchmarks may not align with user needs.
  • A better measurement strategy is needed for evaluating AI systems.

Transcript

0:00 95% Einstein, 5% Mr. Bean, and if it doesn't tell you when it's the Mr. Bean, then how are you going to automate anything? I wrote a document at OpenAI. We could have had AGI last year, and I still believe that to be true. Pre-training is a giant dumpster fire of wasting resources. When I first started this project, I literally thought it would take a week. You can now express things that you couldn't express in software before. I think that the future will be unbelievably sci-fi. We're building prod. Absolutely.

0:32 >> That is Jo Almera, CEO of Typesafe, the company behind Jev. A month ago, nobody had heard of Jev. Then it launched hit a nerve in the AI world and quickly became the most copied model in AI. Jev is both fascinating and perplexing. It's a new kind of model and is clearly a big deal. Yet, even some AI insiders struggle to explain what it is, while others dismiss it as just a classifier. So, we went to the source and asked Joo to walk us through Jev in plain English. What it is, where it's going, and why copycats are missing the point. I'm Matt Turk, and this is the Matt podcast.

1:06 Jo, welcome. >> Thank you. Thank you. Hey. >> All right. This is a super fun time to be doing this conversation in part because just yesterday I believe as we were recording this you guys announced your gigantic series A 870 million series A at a 7.5 billion valuation. So it's been a wild 3 weeks just looking at some stats. you launch video at 40 million views. You talked somebody talked about a million users a day, a trillion tokens a day, a third of the Fortune 500. all of this in in in just a few weeks. So, it it sounds like you're indeed building what you call the evolution. I love the term. I'm just curious what it feels like from the inside.

1:50 >> Where do I start? I will say that a lot of those stats don't really matter for what I feel like is the evolution. to me, I knew it would be happening when builders started playing with it and doing the weird stuff. And especially when people were starting to make apps that they maybe could have done before, but didn't bother with because AI was so expensive. You know, I I we push the boundaries of intelligence per second, but I think the future is going to be intelligence per dollar primarily. And I think really getting people to play with like like the idea that intelligence per dollar has dropped two orders of magnitude is what will make the evolution unstoppable independent of enterprise adoption users stats whatever it to me it's about people playing around with things and starting to I can't think of a non-corny way to say it but like build the future you know like like that to be the unstoppable energy and no matter how big that initial snowball is it will become gigantic.

2:51 So, I've been celebrating this for something like 2 and 1/2 weeks now. I've actually had to do soularching because I think one of my big life goals has been achieved. and yeah, the stats are obviously great. Growing fast is obviously great. I love that we are also getting a lot of the distribution that we knew we would get in two years, but much faster than we thought. These are all obviously phenomenal things. but the leading indicator of it all is builders being inspired to do new things.

3:25 >> And just to ask you a couple of like startup related questions. So you guys were in stealth for 2 years. I'm I'm curious how you thought about it in a world where like everybody's on X every two minutes to talk about something. Why did you decide to be in stealth? >> No one's asked me that, so that's fun. I would say I think quite contrarianly and I think that this whole AI has a gigantic overpromise underd deliver issue and you know I think people really saw that with our launch like when we first launched the video people were like oh that's fake it's kind of interesting but like there's no way it's true. The launch really picked off up when people started playing with it and showing their results. And I think that underpromise overd deliver of our launch is what really made it take off. And it's been I can't think of a launch in in AI that has been under promise overd deliver since chat GPT. So that might be why it took off. I'm not an expert in like social dynamics. So I really really you know for better or for worse I ignore the noise in the field and really focus on what I think is most important. And I just did not buy the story at all that AI was solving all sorts of problems because if you actually look at all the evidence it's AI is incredible and also incredibly underwhelming. And that allowed me to really ignore all of the noise in the market. But you know doing this in obscurity while everybody's you know getting celebrated. How do you keep a team energized and keep yourself energized building something sort of you know without anyone really knowing?

5:05 >> I don't think that I I maybe maybe the last couple of weeks have changed me. I hope not. I don't think I get really motivated by being celebrated on all of that. I actually had to do a lot of therapy just to be okay with being in our launch video because I actually before all of this enjoyed being in obscurity. Like I'm a technologist through and through I think and what really motivated me for this is that there was clearly a gigantic technological gap and it was clearly obvious no one was addressing it. So that was my I I I never had any doubt.

5:41 I it was even when I first had this idea long before I started the company, I wrote a document at OpenAI, we could have had AGI last year and I still believe that to be true because AG the how open AI divides AGI is a very low bar and the models are super smart. like like we're not really putting those facts together and I think a lot of the people who like to commentate on the field just say like oh it's a distribution problem where like that's obviously not the case if you pay attention to any like real world stuff.

6:15 So I had very little doubts. I think that we managed to get a team like you know our team obviously doesn't care about you know maybe there there's a spectrum but like our team is not driven by the hype cycles as well because if they were they wouldn't have joined us. I think they really believed in our mission. Or if they didn't, you know, we had nontechnical people on the team, they really believed we had a bunch of like super smart people. Maybe they don't understand what the model is for, but they're like, "Holy crap, these guys are cooking. This is worth a bet." So, I don't think the team is that affected by it.

6:48 >> Well, what's funny is that I hear all of this and what's fascinating at the same time is that you guys need to seem to be kind of like naturalborn marketers. And I'm sure you you'll cringe at the term, but you know, you you you keep writing, you're tweeting very effectively. It sounds like you're doing this out of authenticity because you're so inspired by the mission and the rest follows. I'm trying to tease out you know, lessons for for others that would love to have, you know, a fraction of the success that Jeb had at at launch.

7:19 >> So, number one, I don't think we know about marketing. we still do not have like a marketing team and I would actually argue that it's not like we're geniuses at marketing. My argument is that it works because our product is awesome. So actually I think our product is carrying the marketing despite what people say and like you could call what we're doing authenticity perhaps like we are just trying to have a good time like our the the blog post yesterday was a fundraising announcement. Someone in our team gave me a draft and I was like, "This is the most boring thing I've ever read, but if you want, you can ship it because I would never read a fundraising announcement anyway." So, you can actually put whatever you want. I don't care. And they were just like, it would be nicer if it was in your voice. And I was just like, I would never I would like why does anyone care about this?

8:10 The product is the thing. The fundraising is tends to be like an ego game. And they're like, oh, you know, enterprises will see you as more trustworthy. Recruits will see you as a more like legit thing. And we're like I was like why don't we just say that directly. Hey recruits we're a real company. We're here to stay. Hey big companies we are going to be building out this enterprise platform. We are also here to stay. Look at who's invested in us. You should trust us. And I think that maybe my intolerance for is helpful there. but I truly believe that it is not a replicable strategy.

8:48 I'll I'll give a fun side story about the fund raise. Some investor said something like, "Holy that was the greatest pitch we've ever heard." And I just told them, "I don't believe you because I've been giving the exact same pitch for years." And no one cared until we had, you know, absolutely world topping product market fit and then they forced themselves to understand it because I don't think they would have gotten the pitch otherwise. So, I believe the product is carrying everything right now and that's perfect for me because that's my style. we knew this would be technological shock and awe compared to what big AI does and that is probably what we intend to keep doing strategically.

9:33 >> What I'd love to do now for a little bit is a very basic plain English version of Jeb. There's a lot of people that have heard of it, but like very few people are able to actually articulate what it is. So, I'd love to start from the top and maybe start with this whole concept of system one. it's it's a reference to something. Explain what it is. >> I would actually like to talk about is machine native AI rather than system one. I actually think that system one is more of an implementation detail to machine native AI and I actually think most people today think of AI in a very particular way like AI is always like the chat box to the side that happens to be the main character in its own little show because it doesn't play nicely with software and I think step one for people is like this is not you know chat GPT++ this is like AI for a completely different use case and is how I would do it and I would talk about you know even the analogy to classifiers I think being called a classifier is a compliment we are not limited to classifiers but understanding the beauty of classifiers I think is a necessity to understanding why what we're doing is very valuable >> all right so machine intelligence is the goal system one is an implementation detail but like system one versus system two that's from the book thinking fast and slow. And the whole idea is that there are certain type of decisions that you can make intuitively and quickly which is system one and there are certain decisions that require elaborate thinking and that's system two is an accurate categorization of where Jeff falls in the world that Jeff would be in the system one kind of world whereas your classic LLM let's call it chat GPT for just as a as a shortand would fall in system two. Is that directionally correct?

11:36 >> It is directionally correct. I would actually add the nuance that I think that models not trained with like the the latent reasoning, you know, like the reasoning where that they keep hidden tend to be system one models and AI is very strong at system one in general. system one in the conanmon sense and it is the reasoning training that makes them capable of doing system two tasks but they are always fragile at system two tasks and my claim is that AI is just good at system one and that is extremely valuable on its own and as far as I can tell we have not come even close to solving the jaggedness of system 2 and I I and then it would be throwing out the baby with a bathwater to say it's system two robust.

12:26 >> Okay, great. If there is a need for an AI that can do system one particularly well, which is a dev world, it means that there's a number of decisions in the world of enterprise or otherwise that actually pretty simple decisions. Is that is that the right starting point >> that that yes though I would go into like a big aside about software as well and machine niveness and a little bit unclassifiers if I could right now. Does that sound like an okay aside to jump into?

13:00 >> Sure. I would talk about not that in business that there's specifically system one decisions that need to be made like yes I think that that's how it'll start but like I would talk about like how much software can do with its basic logic gates like software can do so many things in the world so many things like basically all of technology from like what 1980 to 2020 was driven by software maybe that's an exaggeration but it feels like that to me as a software engineer, so maybe I'm biased.

13:32 and you can do so many things with like these simple logic gates that just do like ands and ores on bits. That's kind of crazy. And it software becomes big and powerful, not by saying like there's an end here or an or here in a business. It's by doing like a unit of work so reliably that it can become like a bigger thing and then composing it together. And my claim is where AI will get incredibly valuable is not people mapping out explicitly like here is work you know here's a job here's the system one decisions let's like do a workflow automation on that I think that's how it'll start but I think that AI will explode in usage once people start taking these you know these workflows putting them into like a box and then now people call the higher level boxes and that is what software is all about >> because that box will be solved and safe and reliable.

14:29 >> Exactly. And you need so many that is what I describe as composability. And and the beauty of software is like once you spend the effort up front to making these boxes, you can infinitely replicate them, right? Like software is so unbelievably cool. Like imagine you have Lego sets where you can build up higher level things, but you can copy paste them as many times as you want. That is what software kind of is for automation. And that I think is how you get more and more powerful properties.

14:58 So system two tasks are not off the table for our models. We just believe that system 2 tasks should be composed from simpler building blocks that are so reliable that they can be trusted to be built on top of which AI so far has been incredibly non-composable. Like it plays nicely with nothing except other AI and that's that's kind of tragic. Yeah. >> Amazing. All right. So this is where this is going in terms of where we starting at a at a super practical basic level. one of the use cases that you mentioned or the team mentioned is customer support which and the claim is that the current ways it's handled with LLMs has not been as efficient as it could. Let's take the example of I'm sending an email. I'm saying I got charged twice and I'm very angry and I send that email to an LLM based system scenario one and then scenario two I'm sending this to a jev powered system. What happens? Compare and contrast.

16:07 >> Can I do an easier example than customer service is actually quite complicated. let me do something that is very similar to customer service but is a lot easier to think about. Like imagine an executive assistant for your email. It's kind of like customer service, but the amount of things you can do is way simpler. And as far as I can tell, I could be wrong that like a scheduling assistant is still not solved. I, you know, I have emailed a lot of people. I don't see automated scheduling systems.

16:37 And like that's kind of weird because it really looks like LM should solve this, right? So people have been trying for years for this. And how you do it in an LLM is you would, you know, make a big system message write up of all of your preferences, blah blah blah, and they would be incredibly good at stylistically being accurate. Like it'll never not look like a well-written thing, but it will often make subtle mistakes, not in the style, not in the psychopansancy or psychopansancy. I don't know how to pronounce that word.

17:08 you know like it'll always be people pleasing because LM's are so good at people being people pleasing but it may not make the right decisions and what people will do is they will like keep programming their system prompt like please do this if it's an investor do that if it's inbound email do this you know blah blah blah so what ends up happening is you're basically programming in natural language this giant system prompt and as the complexity rises the error rates get higher and higher and higher because you're just praying model. Please don't make those mistakes. And what tends to happen is it's embarrassing when those mistakes are made or it's costly. You know, you may not have like scheduled something with an important investor or something like that. And then you need to like rip the system out and toss it away and just use, you know, a human because you know it it's like it's just obvious what to do there and they can like ask you when they're uncertain or whatever else. So LLMs, to my understanding, still have not solved this problem. the I I don't know if anyone has solved this with a Jev like system yet, but I can compare and comp contrast the differences of it. So what you can imagine instead of programming in natural language like describing like if it's important blah blah blah if it's from this blah blah blah if it's inbound blah blah blah you can actually make individual classifiers for each of these things like smart if statements like you can now express things that you couldn't express in software before cuz how can like these bit comparisons figure out if it's inbound or not right like that's an intention and it requires like natural language common understanding and the idea is you can make classifiers for all of these things and you have this legible system where not only can you measure if every if statement is accurate. You know that none of those if statements will be forgotten and if ever a mistake is made, you can add it to an eval because the system is very legible.

18:59 So it'll probably start out less good than the textbased system because a textbased system has a lot of sane defaults. You know, it looks very reasonable because that's what it's trained to do. It's this train to like like look human pleasing, but it actually doesn't have any of the actual behavior you truly care about and you don't have the control over that, you know, and that is a major problem for anyone who wants to truly have something run in the background. They want it to to not worry about it and they want to have control and they want to have like some idea that it's not going to just go completely off the rails. And that is what combining like you know system one models with code will enable you to do because code is kind of like you know the the roller coaster tracks. You don't go off the rails. Code prevents that from happening. And similar to that type safety you know that is what the property that the system one models have. And these things also prevent going off the rails and because they're very legible it becomes way easier to build these systems and just trust that they can run in the background. So in that example of the personal assistant the the fundamental basic concept is that JV is going to take the input which could be data and or instructions and will translate this not into text which is what your language model does but into a set of probabilities. So they will say this is urgent yes or no escalate yes or no and if you escalate send this to this department or that department right >> yes and also ooh there there's like like interesting nuance here where not to just you don't just ask Jev should I escalate because that's a very high level decision like you know it requires experience and trust with working with someone to know if you should escalate or Not the idea is that you can keep on decomposing your questions into like the lowest level.

21:02 you you we have the ability to break it down like oh I want to filter out all inbound sales from my inbox and I want to maybe filter out like recruits emailing me right because like we have like a like an a specific email list for that like that might make sense but there might be a world like oh I've been I filtered out the wrong email let me add new behavior an example is if someone else is on an email I probably shouldn't be on that email but if they say can you look at this? Maybe I should all of a sudden see that. And this is like nuanced smart behavior that requires like a smart person to think about that behavior. So either way a system is getting programmed either by natural language or code. But if you do it by code, it's cheaper, faster, and guaranteed to run. and the idea is you can keep on like drilling down into the behavior you want as you make it more and more and more great for your use case rather than with LLM there's only so much control you have >> and the developer gets a score back for whatever little sort of unit they're focusing on is that email angry or not angry 0.85 85 and if it's above 0.9 then trigger this action. If it's below don't trigger that action.

22:24 >> Yeah. Yes. And also actually your your note on the probabilities is also really really great. There's a reason why we do that and we learned this the hard way. like when we like shipped the OpenAI models they don't have like fine grain controls. And what ends up happening is I I'll use the simplest analogy which is refusals, right? Like how does it make sense that you have a response? >> Refusal is when the model refuses to answer.

22:49 >> Yes, I'm sorry I can't do that. And it's kind of crazy that for some responses you just refresh it and sometimes it refuses, sometimes it doesn't. Like that's insane behavior. And also if you're building on top of this, the your threshold for refusals might be very different, right? Like if you are Disney, you probably have an extremely different threshold for refusals than if you are character AI, right? Like character AI likes to do like roleplay and stuff. Disney is very protective of their characters and IP. it doesn't make sense to just have a discrete decision. The most powerful way to empower developers is to give them back as much information as you can so they can then choose for their application what is the threshold. like, "Oh, I accidentally mclassified an email from an extremely important investor. Let me tune that threshold lower because I'm okay with false positives. I really can't stand false negatives."

23:43 >> And the reason why Jev is much faster and much cheaper and I I know that's only part of the claim and that the bigger claim is intelligence everywhere. >> Thank you. You listen to me. Hell yeah. Hell yeah. >> Evolution is in motion. So why is it able to be faster and cheaper? Is that because it's taking a subset of the problem or is that because it doesn't have to go through the whole next generation than an NLM would otherwise do.

24:14 >> I actually can't talk about that without the intelligence. If I wanted a faster model, I could just chop half the network out, right? And then you have something twice as fast, right? Or or twice as cheap or whatever else. So it's very hard to talk about that without intelligence on its own like obviously we're in a new quantum of speed and because we toss out the very slow expensive part of text generation. So you know we we've talked about that we are obviously not auto reggressive. We much prefer like this parallel you know gradual discovery of what the right thing is. But what I think people don't really realize is not just like the obvious engineering part. You're like chopping off parts that are slow, but the you get what you optimize for part.

25:02 And the we want to extract intelligence in the best, richest, most dense way out of these beautiful cognitive cores that AI already is. So I I you know the tragedy before our launch was that AI is so clearly unbelievable that's so useless for most things. And it's not just that we put a plug. It's not just like tossing out the bad part, but also like smoothing out the intelligence and making it as robust as it can be to surface that intelligence. And it it can't be talked about without the other stuff. So it's faster and cheaper to play back what you've said so far because you've removed certain parts and you polished the the rest. So the the part that you removed enabled you to do this ultimately very specific job very important and very ubiquitous kind of job but but specific job faster because in some ways it's specialized for that job.

26:01 >> There's a bit of an implica oh man I get into my nuance and I love it. I would actually argue that it is not narrow and specific. Like you could fine-tune small language models to be narrow and specific, but I would argue that classification is AI complete. And like, you know, the field has talked about this for a long time. like a general purpose universal verifier or classifier is absolutely AI complete because it's easy to turn this thing into like even if you want text out which I think is somewhat rare for a system you could turn it into that if it was perfect. So I would argue that it's not like narrow and specific. It's just a different shape of intelligence and that is you know it's a it's its own big general branch and there's going to be so many beautiful subbranches of this. You know we have three types right now but we are going to have so many more and we're going to show the world why we didn't call them decision models.

26:56 so, oh, it's to me it's about like how do we use these incredible compressions of human intelligence as effectively as possible and yeah, it's it's beyond just removing the obvious slow parts. >> Okay, great. Let's talk about why language models are not able to do this and why a whole new family needs to be invented to to do this. And so my understanding is that part of the root of the problem is RHF which you're one of the co-inventors of which served its purpose incredibly well for what it was supposed to do. However comes with a set of side problems which in particular are bad for the mission at end. Maybe let's let's start again in an effort to make this approachable by by a bunch of people.

28:01 What is RLHF at the simplest level? >> Ooh, I would describe as RHF as making models chat and making models pleasing for chat >> because so models are pre-trained. So they have the whole all the knowledge in the world. however turning them into something that can converse in a requires them expressing certain type of preference and certain ways of interacting with human right and lots of stylistic stuff actually I think a lot of the optimization for RHF in hindsight looks extremely stylistic rather than is it about you know like being better at like tapping into its facts deep into its network.

28:46 >> RLHF stands for reinforcement learning from human feedback. So how were humans involved initially when you guys did that? >> So actually there's many different parts of RLHF. I think I started working on it perhaps end of 2020. but there was like predecessor work to be clear that I would want to call out. So I think that we built on top of the shoulders of giants and many of those predecessor works could be called RHF as well. I'm happy to cite them now, but I if it's a I'm not going to do it right now. I've cited them in other spots. the the thing that made RHF RHF in my opinion is about the task. The task about optimizing for human preference and the shape of it as being text in text out. That I think is kind of unarr that to me is the root of what originally arlay was. At the time we were still trying to figure out this shape. I think it's very obvious in hindsight, but there was a lot of uncertainty. We were kind of like the black sheep of open AI. and it like no one likes putting expensive humans in loop because it's very hard to iterate on it. And we were really trying to figure out like, hey, will this work?

30:02 Because when we were playing whack-a-ole with the issues, like things kept coming up and we kept having to solve them until we made it like really general and pleasing and good. I believe that the hardest part of creating a new direction in AI is even knowing the task. So if we knew in hindsight the task should be shaped this way, it's possible. It would have taken us u like it would have been way way way faster at this. But because of that uncertainty, it was a a little bit harder to figure out like yes the way to do this is data make this diverse data and also like even the specification for labels like here is how to label these things because this is the thing we want out. These are things that we didn't know at the time which everyone who copied us didn't have to figure out because we had done all of the hard work of that iteration and then published it in a paper. Not to have a stab at them, but I do want to call out that a lot of the hard part is the discovery here. And I think that that's a lot of our hard part with type safe. So we are to our best proxy optimizing for human preferences and that is why you never get outputs that obviously look bad even though they might be equivalently bad to one that is just wrong. like you never have models where you never have outputs these days even though the models are wrong all the time where you can just look at it and be like this is clearly crap because it that's punished so hard and it's like really warps the probability space for these models.

31:29 >> Yeah. So and we we're talking about all of this because again that's sort of the heart of the of the problem. so because the models were trained to please humans and sound confident as part of that process of our LHF then that's why you end up with issues like fency over refusal that that you alluded to >> and also like mode dropping like the models the chef models are very stylistically repetitive you know they they're all repetitive in different ways but you can like kind of get a sense if you use a model a lot like wow it keeps saying this thing you know and that is a side effect of RHF because RHF heavily encourages the models to stay on their on their stylistic lane because that's the safe path to not get punished and this this results in like a lack of creativity that pre-trained models didn't have pre-trained models were like wild and creative and weird and they have like all of the styles in the internet inside of them and they somes just go completely off the rails which is beautiful and they didn't have this particular problem that I think people from what I've seen kind of dislike in LMS.

32:42 >> Great. So again, all great for all of us to discover the power of generative AI through a chat experience, but bad for the problem of classification because you have over refusal, you have secency. Then you also have the latency and overhead problem that we alluded to that because the model was trained to do text to text. then it needs to regressively generate pros line by line and sound very confident doing it which is which is not helpful if you're trying to say customer angry not angry or problem yes yes yes or no >> I agree with all of that though I will emphasize that I do believe RHF makes the models slightly dumber at decisions the you know than the pre-trained models the the the root of the problem to me is that RHF actually greatly increased expectations. So if you imagine like how good the models are from human perception versus like reality. RHF really warped that because all of a sudden people were like holy smokes these models are really really smart. We can do all sorts of stuff with them and that decoupling is I think quite harmful to the world and that to me is was my biggest worry here. So like it you know no one you know like the expectations for pre-trained models were very different but because RHF models are so pleasing to humans and they seem so smart they it makes people think like holy smokes a lot of stuff can be automated and that empirically did not happen.

34:22 >> So that's RHF which is a key part of post training. People may have heard of other parts of post- trainining RLVR in in particular, which is how models have gotten very good at at math or or coding. Maybe define very quickly what it is and then explain why it's not solving the problem you're trying to solve either. >> Oo. So RLVR is short for reinforcement learning from verifiable rewards or for I I I don't actually know that one. But the idea is for simply evaluatable test cases kind of like a math problem or like a proof or something like that where you you know an answer can or like you want to pass unit tests. Can you do a lot of stuff in between to write code to pass those unit tests or to write math to to get that final answer? And a really key part of RLVR is not just the verifiable rewards because people have been trying to do RL on verifiable rewards for many years before RLVR was created. It just so happened that it didn't work. So the the codegen team at OpenAI was trying for quite a while and it just never really worked because you actually needed like this reasoning chain in the middle to do all sorts of creative stuff in between so that you could get answers that like pass the unit tests and the math and all of that stuff. So RLVR also means like the training for reasoning, you know, with this like stuff in between in the middle where you can do arbitrarily complicated stuff sometimes. So the RLVR is a lot about passing simply evaluatable checks and RLVR you know RHF caused a massive overpromise underdeliver based on like human feel of AI but RHF actually started plateauing on benchmarks. but RLVR does not plateau on benchmarks at all. Like RLVR crushes every benchmark because by construction RLVR is kind of the shape of the task. if you wanted to crush benchmarks, you know, like benchmarks are by construction these simply evaluatable things. that is just perfectly matches the shape of that and it causes a different type of overpromise underd deliver there. RVR is obviously very very cool though. I I think actually I think RHF is very cool too. but I just worry about the lack of automation and empirically RLVR hasn't really changed automation.

36:51 >> Yeah. So our our LVR doesn't help us in the problem we're discussing because it focuses on rewarding the final solution whereas what we're trying to do here is probability and understanding yeah understanding >> it's also stocastic it's also like unreliable and like it I think people like the word jagged you know like it is sometimes incredibly smart like the way I like to describe it is 95% Einstein 5% Mr. Bean and if it doesn't tell you when it's the Mr.

37:29 then how are you going to automate anything you know so it's jaggedness and you know non robustness I think is the real issue with it because it's very good at answering like simple classifications too but if it doesn't do it reliably how do you automate anything in the first place and that's kind of where calibration comes in and the models are still not very good at that >> okay great all right so RLHF kind of creates the problem RLVR doesn't help us So therefore you guys had to invent a new post training method that you call RL CD. Is that correct?

38:09 And if so just explain as much as you can share what it is. I don't know if I would call it a post-training method. because like you know when we made RHF the concept of post-training did not exist and you could call RHF a fine-tuning method but that underscells how much change you did to the whole process. So like no one would just call it fine-tuning even though maybe there's some technically correct way where like that's 80% accurate. So I wouldn't necessarily call what we're doing post training. I think we do a lot of weird stuff. I think that RL tends to be again a implementation detail of the whole task. You know for RHF the task was not just doing RL and PO and whatever. It's about figuring out here's what the inputs and outputs should look like and here's how you instruct people to do that and here's what our north star is and here's how we balance the data sets to get a really good model for this thing. And for RLVR you have similar stuff. RLVR is nothing without its like latent reasoning. You know, it's reasoning chains and then this enables you to have like a new mixture of data sets. How you construct the data sets and even what difficulty of data set you want completely changes with your base models and shape and all of that. So, the RL tends to be like a like a like a small slice of the whole task pie. and you know, we did have to make a new type of training, you know, that we do call RLCD because no one has done anything to optimize for machine native intelligence. But I just I don't want it to sound like if you found the right RL algorithm, it just magically gets solved because so many things go into it beyond just the RL.

39:54 >> So to be able to do training, you need to a bunch of data. This is data for a very specific kind of task in the new paradigm that you're inventing. Where does that data come from? >> So I'm going to use the analogies in the past because I think that that's an effective way to answer this. Before we made instruction following a thing, the data for instruction following literally did not exist in the world. Like no one had like here's a human answering simple strings. No one had preference data from like outputs of LMS to see like which one is better. So this straight up did not exist in the world and in creating the task we had to create the shape of the data iterate on it and also collect that data. Similarly for RLVR the data in RLVR tends to be environments right like these RL environments to like like try stuff with like sandboxes and tools and whatever else and those environments kind of existed before RLVR was created but they were not useful and there was no market for all of those data sets because no one could train on them. so similarly for us we needed to like create a new type of data a new shape of data iterate on that and then actually create the data itself. So by construction for all of these things these always need to be made inhouse and they always start inhouse because that is how you even iterate and think about the shape of it. So we have done all of it inhouse and that as far as I can tell is going to stay the case for quite a long while because you know the world has no idea what it's doing right now and all the world experts are our team. So we just make it ourselves.

41:38 We obviously don't need user data. None of those things do. It it always requires a lot of care for the task and really understanding what the northstar is. >> You had a a great post recently called the bitterest lesson in in relation to Rich Sutton's u you know extremely famous essay. walk us through what that is and where our LCD fit. are you you know you data compute algorithm where where where >> I would actually argue that all of the great methods of training fit in the exact same way which is they are novel tasks. I'll walk you through it first though. So Sutton's bitter lesson roughly paraphrased is that compute and scaling matters more than algorithms. So when people are like bitter lesson scaling law etc. This often means like why does it matter what algorithms we do? let's just scale it up and throw more compute at the problem. I would argue that this is an oversimplification specifically for the kinds of RL environments that Sutton used to work on at the time. He RL is often done in games and in games there's no question about task and data because these are all functions of compute. But in the real world, you can't really do anything with compute without data. You know, like you can't continue to scale pre-training on the internet because we've run out of internet data. and data is like obviously the most important thing there. But in order to even know what the data should look like is you need the task. You know, you need to know the shape of the task. You need to understand it. This task needs to be linked to valuable things in the real world. And the simple way of explaining it is you get what you optimize for.

43:22 Everyone seems to intuitively understand this except for machine learning researchers because they don't like to think about that. but you get what you optimize for and you need to know what you optimize for and once you do know that you can then make data scale compute and have fancier algorithms. So for RHF is to me a task you know like making models good at instruction following in chat is a task itself. It has its own shape of data its own algorithms its own compute etc. But the biggest bang you get is always from even figuring out the task in the first place. You know, chat GPT to pre-hat GPT was the largest jump in chatbot intelligence ever because it was kind of like an adult, you know, battling toddlers, right? VR too. You know, Qstar and 01 were like on a league of their own, right? Like it the the it wasn't they weren't really used for anything useful in the world, but they were truly in a league of their own in terms of benchmarks, right? that that that's incredible. It was like the code of like the adult to the toddlers. And similarly, because we're the only ones optimizing for machine native intelligence, it is all of them are the right. It is kind of like the adult versus toddler situation where we're in a league of our own, at least from like a reliable machine native intelligence perspective because we're the only ones doing the whole thing.

44:40 >> And you you don't do pre-training though, right? Like do you where where what what is the basis for Jev? Like do you use like open weights models? >> We do use a combination of models. It's like pretend models have a lot of like compressed intelligence in them. They're very text shaped. So we need to do like some disgusting things for it. That's why I don't say what we do is post training per se because again these disgusting things I wouldn't call it post training as a someone who likes post training. It's kind of a bastardization of post training. I don't think post training people would want to be associated with the terrible terrible things we do. But there's just a lot of intelligence like in the variety of openweight models that we can you know link up and MacGyver into this great intelligence. so we do use that.

45:32 you know everyone knows that we raised less than $40 million until recently. So, it's either we've had like an absolutely tremendous breakthrough in pre-training or we just are more resourceful than other people. I think it's the latter but we did not do pre-training. And honestly, I think pre-training is a giant dumpster fire of wasting resources. So, I hope to never do so, but who knows what the future will bring. >> So, the existence of that pre train model is an interesting segue into the whole classifier conversation that you alluded to earlier. There's a number of people when Jeff came out that said, well, it's just a classifier. and for context, a classifier being effectively an an older form of machine learning, you know, pre-generative AI, things like XG boost that have been used for many many years for things like email spam classification. In what way, and I know you embrace the fact that you're somewhere a classifier, so in what way are you a classifier? And in what way do people that say that you're a classifier miss the entire point?

46:37 >> ML people especially researchers really miss the value of DEV still and that's really good for us because it's not fancy ML and it's just a classifier is among the dumbest criticisms in my opinion because classifiers again are classifiers are not a form of old school machine learning. you know certain types of classifiers are but classifiers are an interface they are the shape of putting intelligence into software systems that's useful right like how does Facebook work classifiers how does Tik Tok work classifiers how does Google work classifiers it's not like a bad thing at all because in order to make software more powerful software can do so much with it it's like its like dumb little bit comparisons is you want to put your intelligence into branching in your code in in order to do complex behaviors and the way to do that is to influence the branching via you know intelligence which is what classifiers literally are. So classifiers what we're going to do is not only classifiers but we've started with classifiers because they are super duper useful. You know, it's a it's an incredible compliment to say it's a classifier. And actually, this would be not a fair way to describe Jeb because it's too much of a compliment if we said that Jeb was equivalent to anyone having like a 20 person machine learning engineering team from 2019 to build every single thing you think about on the fly and it's instantly done. But actually, it's smarter than that because we have better models today than we did then. And like I don't know if we're there yet, but that's a hugely awesome compliment.

48:18 And machine learning people don't get that because they're too busy like trying to do like the the the least valuable part of the bitterest lesson, which is algorithms. And they keep trying to like build their god in a data center rather than solving real problems and see the real world and see how little automation truly is getting done. And to the God point, you have like this awesome saying which was actually also on the t-shirt of your launch video saying prod not not god which >> we're building prod not god. Absolutely.

48:46 >> A fabulous way of of of putting it. and to unpack some of this. So your typical classifier from 2019 was also focused on a very narrow task and what what you're saying is that effectively because this sits on top of a pre-trained model that has all the knowledge in the world, you're basically able to apply classification to any task. Is that is that fair?

49:18 >> Any task any task that the LMS can do, right? And that's why we call them system one models because they are good at system one tasks, you know, or they're adjacent to that. >> But you don't need to train a an email spam classifier because the the pre-trained model has all the knowledge in the world about about emails. >> Yes. And you know with a decomposition like you you might not even want a spam classifier because that will be the same for everyone. You might want your own thing where I want to say emails from my chief of staff even if she writes like AI should be marked as not spam. So it like the programmability I think just unlocks so many more use cases even if you had the equivalent of every engineer having a team of MLES machine learning engineers like making classifiers for them. So I think that there's like a real real beauty to the generality and programmability of it.

50:11 >> Yes. And to make sure we cover this very very important part of the the whole discussion, you mentioned programmability. you mentioned composibility earlier. We talked about price and speed. and basically the the the fundamental claim is that if all of this is is is true and happens and you guys are very much building it, then it can be ubiquitous across software. >> That is the dream I hope for. Oh man, I have grand hopes that I think are not even that unrealistic. I think that so much could change in software like you know look at how much was done with bit comparisons you know we have been building like so much valuable sick stuff like you know just look at 2019 software has like I think made the world a lot better like sure there were some misses but like we've automated so many things via these simple like sticks and stones comparisons and imagine If we can just expand that more, more more more more, I think that I think that like we'll be able to automate so many freaking awesome things. I think that the future will be unbelievably sci-fi and we're going to do whatever we can to like unearth the intelligence that is like robust enough to make it even more sci-fi. And that means again like to play it back it could be it could be intelligence like in every row or column of a database right so it's it's just embedded intelligence across all things software >> absolutely across all levels of software too and I think that with this stacking as you make you know like this valuable box and then replicate it and if you make it fast and cheap enough now some other box can call this thing 100 times and now you can do higher and higher units of cognitive automation work and who knows what those things are going to be but I think that there can be like a lot of insanely cool things yeah and that's flipping on its head or in some ways a complement to everything that have been done in software and AI to date right so the the cursors of the or the cognition have been very good at accelerating the pace and the the how easy it is to create software but what you're saying is is is It's it's that you can embed intelligence in software.

52:36 >> Exactly. So the great question the cursors and cognitions focus more on writing the software itself faster and cheaper presumably. while what we want to be instead of like kind of being like a so what they're doing is being kind of like a software engineer but what we are trying to be is more like a logic gate you know like we want to be inside of the software that expands the possibilities that software engineers can make rather than you know replacing the software engineers or anything like that.

53:05 >> Great. Let's talk about all the copycats quote unquote because that's you know a topic that's top of mind on X and everything. So for anybody that hasn't followed, so you guys have been out of stealth for 3 weeks. and then in the 3 weeks you've had basically every everybody and their brother come up with with something from Cloudflare to data bricks to I think Microsoft just last night Satia announced decision one category. What do you make of that? And what do you say to the naysayers?

53:37 >> The naysayers of us. >> Yeah. the people that think that you were just completely copied by everyone in 3 weeks and therefore there's nothing left which obviously is not the narrative but I'm curious like how you position it. >> I I think we will really see like I don't want to punch down too much but I will like describe my take of things. when I first started this project, I literally thought it would take a week like I and I ended up with something very similar to what people are doing.

54:09 But the problem is it was really really clearly not robust. And that was something that has like plagued me for years. and I get the sense that this is a thing that and to be clear I'm a machine learning researcher. So all my self-deprecating comments, all those deprecating comments apply to me. I was an arrogant person who thought algorithms are the way to go and like, oh yeah, I can just, you know, do these shape rotations myself and make this happen. so I was able to make things like this in the first week of this project as well. And the problem was, at least in my experience of it at the time, is it was so unbelievably dumb that it was embarrassing. Like getting the shape of something and and like copying an existing interface is very very easy. similar to what Google did with Flan back in the days of like RLHF, like we actually had an early alpha. They actually published before us. They made like a version of it and it was not just bad, it was worse than nothing. so things like this can happen all the time. And I I I'm curious about measuring how good they actually are from like an intellectual perspective. As far as I can tell from the field, they probably are the equivalent of years behind us in terms of like actual reliability. But I want to measure this before I like say that to anyone else.

55:33 >> And you're not a you're not a fan of benchmarks either in terms of like measuring benchmarks are useless. >> What was your post again? Benchmarks. Liesen lies and benchmarks or something like that. Yeah. So benchmarks are like like they really really hurt users and they actually hurt most the users who they think they're meant to help which is the users who do not have the time to play with things themselves like because they are the ones who are most misled by these. So I get the sense that everyone else is really optimizing for benchmarks and that is I believe probably a bad strategy but again we have better ways of measuring intelligence so we plan to do this.

56:15 my guess is that they are dumb and non-ro, but maybe that will be sufficient for some people's works. And if they solve people's problems, I am super jazzed for that. Like actually, I'm inspired by solving people's problems. My only fear from this and benchmark maxing is that it might convince people that this entire category of models doesn't work because they use one of the dumb models instead of our model. And my request for people is like, you don't have to believe in us. just try our model too at least and stick with what works. because what actually there's like a lot of nuance to making intelligence reliable that machine learning researcher types who really want credit and all of that truly do not understand. and that's why we don't hire those types.

57:06 >> Yeah. say, you know, playing playing back this conversation in in in my mind that look like there's a lot of smart people in the community for sure. You know, I mentioned certain companies obviously OpenAI released their own product called decisions just just just a few days ago, but playing back this conversation in my mind it seems that what you guys are doing is is very fundamental. You know this whole discussion we had around RHF and why that created a problem and how you removed it and replaced it with something and your RLCD and you need specific data for that that seems like a lot of work. So what did people do then in this two weeks when you say interface they just took the results of a regular NLM and removed the text and forced it into some kind of probability but they didn't do the work to get there. just remove >> or to get it to be reliable. And if this solves people's problems, I am super duper jazzed for that. And actually, I do want to shout out Open Eyes decision API for better or for worse. They did not include benchmarks. So, they are a lot better than everyone else in my books. So, in ideal world, they don't try to no one tries to mis mislead users, but some companies need the attention and hopefully they don't create too much ill will. And at the very least people try our models as well so that they can figure out what actually works for them. But if the intelligence and reliability that we have is not necessary I think use the other models that's great you know that like I want problem solved and I want automation to occur.

58:37 >> Cloudflare's version is open source. Is that something that you guys are thinking about? >> We are thinking about it. We've had this debate a lot. The issue is that it's as so as far as we can tell, our model is by far the best. We think it's very far from where we think is like totally ready for everything. And we think that having an API allows us to iterate much much much faster and serve people's needs way better. So we have talked about it, we have thought about it. I would I I we've not really committed to anything, but I think something like that could definitely be in the cards in the future, especially as we go into smaller and all on device and all of those things. Like I just want reliable intelligence everywhere. And I just think that open source people love open weights. I love open weights too, but it's not that it's not the easiest to get everywhere versus an API is. And an API is way easier to give people the vitamins that they don't think they need. And as far as I can tell, open source, at least in the short term, will be a footgun for people. And I'm down to give some people footguns. Some, you know, software engineers love foot guns.

59:52 We we think a lot about how we would want a company to serve us as engineers. and given all that we know and that is just not in cards right now. Yeah. >> Okay. Great. Fascinating. Let's talk about type safe the the company. so in terms of what you're building and what the road map is and and and all those things. So first of all and and realizing that you know you're very much in the in the building phase at the same time you you're you're generating tremendous revenue. I I read somewhere that the the claim is that you're not going to capture all of the LLM market, but you're going to capture a fraction of it and that this is very much based on the Jevans paradox, which by the way is the origin of the of Jev the name. Do do you want to unpack that for us? Like how do you view the sort of economic reality of the of the company? How do you how do you go from here?

60:47 >> I see them as completely different markets kind of. I think in the short term maybe there is like some some overlap in market because there wasn't really a shape of general classifier for workflow automation. So everyone was using LLMs for that. so I a lot of our usage right now is people just switching over but that really shouldn't have been LLM's markets in the first place. You know it is just tragic because that was the only tool available. So there's going to be like an intrinsically system one model market that I think will like continue to grow doing awesome stuff which will be completely separate from the LM market.

61:26 So I think that LMS will continue to be huge for generating text, generating code, all of the above. Like will it go up and down over time as new breakthroughs emerge? Absolutely. I hope there's lots of new breakthroughs especially on the realm of token efficiency. But that is me from like the outside and the sidelines and just cheering on science's progress. So I hope that this that stuff occurs. But I just think that this whole makes software really powerful is a completely different market than LM because LMS are just so bad at being inside software.

62:00 And I obviously think that that market will be huge because I think that you know the analogy to logic gates makes me feel like like intelligence will our our our particular flavor of intelligence will jeans paradox and I think it will jeans paradox a lot more like semiconductors and silicon did and continues to do more than for example like gas does. You know, I think as we find more and more stable use cases to automate stuff and like get successes and deploy it, people are like, "Holy crap, there's a gold mine here. Let's do more and more and more." And as we make it cheaper and cheaper, people are like, "Holy smokes, now this old thing that was producing value, we can, you know, wrap it in another for loop, run it 10 to 100 times more, and create all sorts of value for the world in that way." So, I'm so excited about that. I'd love to finish on a sort of like high level note. So you hold two unusual positions that not everybody holds. One one is that we are not on the path to RSI.

63:02 So AI improving itself. So we're not on that path. but on the other hand, you said that AI has now averted the winter that you thought might happen. So walk us through each and sort of your general take for where the world is going in the you know next whatever year or two. >> On the RSI comment, I think I've just been in the field long enough to see this dance happen many many times.

63:33 People see small percentage gains and then they extrapolate it to like the absolute moon when the reality is Amdall's law occurs. People used to give the exact same story of RSI during the neural architecture search days. This is when you use neural networks to make better architectures and they're like, "Holy crap, it's 10% more efficient. What if that more efficient thing then like discovers the next thing and then we repeat it until we get unbelievably more efficient architectures and that entire field ended up getting thrown in the trash cuz it was never useful."

64:07 people don't understand diminishing returns in the real world. And I think that it's going to be a similar thing for the bottles of today as far as I can tell. Obviously, I have error bars in my predictions. But this keeps on happening despite, you know, people relatively young to the field getting jazzed about their first taste of self-improvement. so I don't I don't think that that's really happening based on the evidence I've seen and based on my sense of where the field is going. I do think that AI is an incredible technology and we are like underleveraging it to absolutely insane degrees and I just think this is like just a completely different branch and I think that AI winter may still happen in the sense that you know the the stock prices of the giant companies and the hyperscalers whatever might crash down. I don't think the hyperscalers will, just to be clear, but I think that there could be some market correction because everyone is pricing in some sort of god moment. so that could be happening. I'm not an expert at this, but the way AI winter I'm worried about is AI continuing to be so unbelievably useless at automation and I just think that that's not going to happen anymore. this whole like and to be clear, I think that AI was still valuable but just not an automation.

65:28 love chat GPT, love quad code and codeex. and still I don't think that that is enough given how incredible AI is. And I think almost everyone deep down feels this way. Like chatbots are kind of feel magical and it's kind of weird how little they've changed the world. So you'd expect, you know, if you bottled up this intelligence and made it super cheap, you could put it everywhere. Yet somehow every single year you get like a famous failure of a company failing to automate a drive-thru because the chat bots are not reliable enough. Like that's kind of like a wild state of affairs and like that that is going to be addressed quite quickly in my opinion.

66:08 >> Okay, wonderful. Jo, it's been amazing. I know you are working insane hours by the way. We're recording this on a Saturday which I guess just another day in the 996 world. >> I will say that I don't I don't consider us a 996 company. It's just a kind of a crazy time. I think that we are as a company quite adult and reasonable. and we have a very different north star than that. and I just wouldn't it's just a crazy time these days.

66:38 >> Fantastic to hear a voice a voice of measure in in this world. So, thank you so much. Really appreciate it. congratulations on starting the evolution and excited to see what you guys build next. Thank you so much. >> Yes. and thank you for being part of it. And in fact, thank you to all the people who were early believers. I will say this at the end of everything because we just didn't realize there were so many people who had the same kinds of problems as us. And I think that still most of the people who've heard of Jev don't get it. And that's okay because the people who really need to know about it know about it because the people who do get it were like screaming in joy on, you know, social media into their friend groups and private chats and everything that like AI is finally a little bit more sensible. So, shout outs to those people and we at Typesafe are eternally grateful to you all.

67:34 >> Amazing. Thank you. Hi, it's Matt Turk again. Thanks for listening to this episode of the Mad Podcast. If you enjoyed it, we'd be very grateful if you would consider subscribing if you haven't already, or leaving a positive review or comment on whichever platform you're watching this or listening to this episode from. This really helps us build a podcast and get great guests. Thanks, and see you at the next episode.

Summary

Jo Almera, CEO of Typesafe, discusses the launch of Jev, a new AI model that has quickly gained traction in the AI community. Jev is designed to automate tasks more efficiently than traditional language models by focusing on system one tasks and utilizing a unique approach to classification and decision-making.

- Jev aims to revolutionize AI by combining machine intelligence with software, enabling more reliable automation.
- The model has gained significant attention, achieving rapid user adoption and a substantial funding round shortly after launch.
- Almera believes that the future of AI will be defined by intelligence per dollar, making advanced AI more accessible for developers.
- The model is designed to be more interpretable and composable than traditional LLMs, allowing developers to create tailored solutions.
- Almera critiques the reliance on traditional training methods like RLHF, arguing they lead to overpromising and underdelivering results.
- Jev is positioned as a general-purpose classifier, capable of handling various tasks without the limitations of conventional models.
- The company aims to embed intelligence across all software, enhancing productivity and automation capabilities.
- Almera expresses skepticism about the concept of AI self-improvement, suggesting that the focus should be on practical applications and reliable automation.

Questions Answered

What is Jev and why is it significant?

Jev is a new AI model that has quickly gained attention in the AI community, becoming the most copied model shortly after its launch. Its significance lies in its innovative approach to AI, which some insiders find difficult to explain, while others underestimate its potential.

How will AI evolve in terms of workflow automation?

AI will become more valuable when workflows are encapsulated into reliable, composable units. This approach allows for the creation of higher-level functions that can be reused, leading to more powerful AI applications.

What is RLHF and its role in AI development?

RLHF, or reinforcement learning from human feedback, is a method that optimizes AI models to interact more effectively with humans. It focuses on making models conversational and stylistically pleasing, although it has limitations that necessitate the development of new AI families.

Why is in-house data creation crucial for AI development?

Creating data in-house is essential for developing effective AI models. It allows for tailored iterations and a deeper understanding of the tasks at hand, ensuring that the models are built on a solid foundation of relevant data.

What are the limitations of AI benchmarking?

Current benchmarking practices in AI can mislead users and do not accurately reflect the reliability of models. The focus on benchmarks can detract from the actual performance and usability of AI systems, leading to a misunderstanding of their capabilities.

© transcribe · For agents Built with care and craft by Gokul Rajaram