LangChain CEO Harrison Chase on the infrastructure bet he got wrong

0:00 I actually think when I'm talking to candidates and all they can talk about is using agents to run and do their job, I actually think that's a bit of a negative signal. >> Why is that? >> I think there's things that agents are good at and there's things that they're bad at. And I think the focus on tools more than outcomes is is not a good thing. My general philosophy has been like build up kind of like a team or a process or an ethos of just iterating super rapidly rather than like being super dogmatic and sticking to like yeah this is kind of like the way it has to be done.

0:25 >> If you look back now at starting laying chain, would you do anything different? I think we built a gateway way too late. How similar do you think kind of like coding agent harnesses are to like a good harness for browsing the web? >> What I have found is that code creates you know memory or creates skills that can be referenced in future agents and for browsing tasks which are highly repeatable. That's really beneficial. What are some of the second order effects that you've seen from like lang chain or agents built on lane chain and like how have you oriented your product around that?

0:58 Hey everybody, I'm Paul Klene, founder of Browserbase and we're here on Navigators, our series where we meet with AI leaders navigating the future of what's happening in AI. Today I have on the show Harrison Chase, co-founder and CEO of Langchain. Now Langchain is one of the companies I first used when I was building with AI. It was instrumental to the way that you could build agents before they were called agents, but as a lot has changed since then. So I'm excited to talk to Harrison about all this stuff. Harrison, thank you for joining with us. Thanks for having me, Paul. Excited to be here.

1:26 >> Tell me more about Langchain, what it was when you started and what it is now. >> Yeah, so Langchain started as a side project. So I was at I was at my previous job, knew I was going to leave, didn't know what I was going to do. And so this was fall of 2022, was going to a bunch of meetups and talking to folks. And as I as I was talking to folks, saw that a few of that this was this was actually the time around generative AI, specifically the image stuff. So stable diffusion had just come out and so everyone was doing stuff with that but there was a few wacky people doing things with LLMs. and so I saw some of the things they were doing and it was like early days of like rag. I think there was a few very basic like math or computer use things and so saw some common patterns, put that in lengchain which was an open source Python package. Tweeted it out. I I still remember Karpathy liked my tweet and that was like a huge victory. Yeah.

2:14 and then just kind of kept on like not you know didn't have any like grand illusions with it or anything. just just was doing it for fun and and then kept on building on it and tweeting on it about it more and a month later chat GPT comes out and it becomes pretty clear that everyone is going to want to build some AI experience. and and a lot of the AI experiences that we were focused on were were called agents. and I actually distinctly remember over Thanksgiving break that year we were trying to figure out how to name a class in Langchain and we called it we called it like agent exeutor and the agent part was right. the executive part was wrong.

2:49 That was a terrible part. But we, you know, we we were thinking of these things like LLM's running in a loop for for a while early on. and so yeah, launched that as an open source and then started a company around it a few months after that. So ended up leaving my job and >> we actually met when you had just started that company. super on I caught you at hackathon and I walked up to you because I found a bug in lane chain. I think you had a PR. Hey, can you merge this thing? So the community support around lang chain from the very beginning it was super viral and it's becoming a little more common place but that was like unprecedented amounts of community love around a framework that you built which is a huge accomplishment.

3:22 >> Yeah I think one of the so one of the things that I distinctly did was again like I wasn't intending to start a company around it. I was just using this to explore the space and so I was also using it as a way to just like connect with old co co-workers and other people and and and and yeah build with other people. So Sam Whitmore who I think we both know who who started a startup and now is at cursor she was one of the main contributors to early kind of like lang chain and I and and id worked with her at kencho and it was a fun kind of like you know reconnection there and so yeah like I think whether it was people that I already knew or the whole community absolutely that was a big part of early things. I look now and I I I think open source has actually changed a bunch for better or worse because of because of a few things.

4:07 one I think yeah there are these just like massive spikes in popularity of these projects. so autograph is an even better example that came out a few months after lang chain and was like the most popular thing ever and then you see a similar thing with like open claw. and and so I do think they're they're I wonder if it's because like coding's become easier and so you have more people doing it. I think you see these just like massive spikes in inflections in these projects. But also if you look at a lot of these projects now I I actually don't think the same sense of community is there because you have all these AI generated kind of pull requests or issues that may or may not be good and it's really hard to parse through and and I know projects including ours are trying to figure out how to deal with with with this influx of new AI generated again PRs or issues and so I do you know I I I look back on the early days and I do think things are a little bit different. I I do think in projects today there's a little bit less of that sense of community.

5:01 >> Yeah. And so you took that initial framework and it's so hard to build this framework these technologies for AI because it's AI was changing so much. So I imagine you started with this thing organically out of the lot to it and the models kept changing underneath you. How did you turn this into a company and how is lang chain the framework changed over time? What you think about lang chain the company also growing over time?

5:23 Yeah, it's it's a good question and they've definitely both evolved a lot. So maybe like going going kind of like in a timeline view, you know, we launched Lang Chain October 2023 2022. We started the company in like January or February of 2023. And from that point we were working on our first kind of like commercial solution which is Langmith. observability and eval for agents. And and the thing that we realized that actually has guided basically all of our our products since then is that like the hard part of building agents is getting them to do what you want reliably. like it's it's the the context engineering, the harness engineering, the prompting. Like sometimes they do it, sometimes they don't. What context do they need to have? What tools do they need to have?

6:05 And the right way to do that has changed over time. But everything that we've built from the frameworks to evalability has really helped with like letting engineers try to make these agents more reliable. So we started building eval observability. So it's not open source. This was kind of like our commercial platform. we launched that GA in in summer of 2023. >> And then one of the things that happened is as you noted like things were just changing really rapidly. like early days it felt like a lot of the primitives for building agents were kind of like quicksand underneath our our feet like stuff was just changing and so in 2024 early 2024 we launched a second framework called langraph and so this I think the trend that we saw was that again to this to this theme of people wanting more control over the agents that they that they built people LinkedIn was too high level for people and so they wanted to go lower level and so langraph really emphasized like low-level control like there's no hidden prompts there's no hidden like cognitive architectures you have full control over everything and so and so we did langraph 2024 kept on doing observability and eval and then in in kind of like 2025 I think things have kind of like actually stabilized a little bit in terms of how we build agents specifically I think this idea of an agent from the technical point of view has always really been like just run an LLM in a loop and let it call tools that was the core idea of of autog that was the core idea of some of the agents in lang chain the issue is the models just weren't good enough at that point in time, but then they got good enough. And this really started to happen in in early 2025. and so Claude Code is an example of when I think this first started really working in parallel. We also started to discover basically all the tricks that would go into harnesses. So giving it access to a file system and using that to like write plans and like offload like large tool calls and things like that. Like I think a lot of the primitives that come with coding agents can actually be generalized to other more broad types of agents as well. And so I think in 2025 the the the the kind of like ground stopped feeling like quicksand and and we were able to start building more on top of each other. And I think that that's when we released that's when we did a big rearchitecture of the open source. So we rearchitected lang chain to be on top of langraph. We then built deep agents which was an even more opinionated agent harness on top of lang chain. We kept on building lang with observability and eval. And then the last like six months or so again as these harnesses have become more stable the infrastructure for a lot of these harnesses has also become more stable.

8:26 And so we've started to build into Lang Smith our commercial platform things like deployments and sandboxes and a gateway and a no code agent builder which really kind of like bring together the the harness and the infrastructure needed to to run it. And so that's kind of been the progression over time. Basically initially 100% open source then add on observability eval. And then very recently in the past six months or so more of this like runtime deployment aspect.

8:52 >> Yeah, it all seems so organic and it's the same approach that we've taken at browser base starting with you know infrastructure for the browsers and like a framework to control it and then like more layers on top. I often ask myself this question. I'm curious what you would think if you look back now at starting laying chain and the progression of products and layers that you've built. Would you do anything different or you think it was the right order?

9:11 >> I think there's some things we missed for sure. I think we built a gateway way too late. I remember talking about this as a team a year year year and a half ago. and and we just didn't really know like what we could provide that was different on top of like a light LLM or an open router. Although I think light LLM was really like the main thing at the time. and I think there I I think clearly there's one of one of the things that I've just consistently underestimated is just the demand for just inference that is out there. Like it's massive. the other thing by the way is just how big and general purpose coding agents would be that market also massive. but yeah I I I think we should have built kind of like a gateway earlier on. I think that is a core building part. I I think that's probably one thing that we did a little too late and we should have done earlier. I'm actually curious for your guys' kind of like journey through things as well because I know cuz you guys have also kind of like balanced the line of like building a harness for browser agents versus building like the tools, right? Like I think you just did stage hand v4 which was really focused on the tools and letting people and correct me if this is off but like more focused on the tools and letting people build their own harness and I think previously you had your own harness more. I'm curious how you've thought about that.

10:22 >> Yeah, you know it's it's funny. I think like every company in this space in the infra space for AI we probably nail the primitive like in our categories and then like you try and poke at the edges of the primitive and find out what's right or what's wrong or where we've built too high of an abstraction or too low. So browserbased like the core browser offering browser and cloud pretty much nailed that like that is necessary there's a lot of challenges with it so that's the core and then we had a framework on top of it stage hand which opened up basically a a SDK to control that browser and the early reason we built stage hand was actually like oh we could just bundle in the prompts similar to how lang chain had bundled prompts to make it easier to translate a action from a user like click this button click the buy now button on the page to the actual command but as models have gotten better and better. Your prompt abstractions over the actual context which in this case for us is like the web page has become like an unnecessary layer. So we still have that for legacy users who actually really like it who've like added semi-determinism into their code bases.

11:23 Our more and more advanced customers are like no just give me like the best possible version of the tool to move the mouse to this point like really optimize the tool layer as opposed to offering like layered sub aents. I think sub agents as like a idea has proved to be a little more challenging how they're actually implemented. Sub agents need to own like complete ideologies of a task not like execute a single part of a task. It's very small like we'd have a sub agent to click that button to save context. So it just the tools and techniques of how to build great agents or agents that can call systems has evolved so much over time and I think as a founder I'm sure you feel the same way. It's hard to pull away from the ideas you had in the past. You have to constantly reset your priors on what makes sense. And that's been something that's always challenged me as we've been building. And as I look towards model capabilities, the bets that I've been making right now have always been like I believe model capabilities will converge on human capabilities.

12:17 Therefore, like this move to the file system where agents are going to have their own file system, it just actually makes a lot of sense because that's what people will do. And if models are converging on people's capabilities, you probably want to master the people tools because that's what they're rled on. So, it seems like the market has gone full circle from being let's build a completely bespoke thing for AI agents to oh, let's just give them the tools that we already have because they're actually learning on all of those things.

12:40 >> This is something I actually struggle with a lot because I on one hand I agree with what you're saying and I think it's very tempting and very easy to kind of like anthropomorphize these LLMs and be like, "Oh, if I was this LLM, this is what I would want." But they are like just different things and so I don't like what is the right like level to to do that at? I I genuinely don't know.

12:59 and to your point, like maybe if URL on a lot of these kind of like tasks that are labeled by humans, it starts to converge to that. but you know, these LMS are great at things that humans are not great at and we should we should keep those and we should lean into those as well. >> so some that's something I'm sort of I'm actually so b like going off of the file system thing that you said. How we were talking about this a little bit earlier, but how similar do you think kind of like coding agent harnesses are to like a good harness for browsing the web? Did you learn things from there?

13:25 Are there things that you think are just different and need to be added in in some form? >> Right. Code code mode or agents that write code to execute tools are called APIs. I think that's a great counterargument to the, you know, humans. Agents will use tools like humans, right? Because, well, they're just going to execute code. They're more like compilers or operating systems. And that narrative has proved to be really, really good. I think that agents writing code gives a lot of flexibility. But there is like that autonomy slider of like do we trust it to write all this code? Can it write code repeatably?

13:55 Should it reuse that code? What I have found is that code creates you know memory or creates skills that can be referenced in future agents and for browsing tasks which are highly repeatable. That's really beneficial. So agents that write code to control a browser is actually the approach we took in our most recent framework stage MV4 because we're like well agents will call our framework almost more like writing little scripts to do these tasks and you'll compile bigger and bigger scripts over time and that's why our recent framework doesn't have any really models integrated at all. It's all but the tool layer. So code mode seems very relevant for agents doing repetitive tasks. I think a lot of agentic work is repetitive. Like when I talk with enterprises, they actually have like a you know a document that says here's all the steps I wanted to do with some fuzzy parts, it's very rare that I see just in time aentic use cases these days where the agent is doing something completely brand new. I think trying to bucket all the agents down into one certain type of agent is very challenging. And I've always thought about like domain specific agents like agents that browse the web, agents that do software engineering, agents that do deep research. I'm curious, how have you thought about the different domains of agents and then working backwards to the architecture they actually should have?

15:07 I think different agents that do different tasks need to have different architectures. >> Yeah, I mean I think this is this is the core use case that we think about enabling is basically how to help people build agents for their product or for their domain or for their use case. I think the the answers have have changed a bunch over time and are still changing. So early on when we built Langraphph, the idea was that you'd create a completely custom cognitive architecture for each thing that you do.

15:31 You you'd lay out kind of like kind of how you were saying like there were these processes and and there are like these just like yeah different steps or or different you know sequences of things you want to do and so you'd lay that out in a graph and then there was some fuzzy part and so you'd put that in an LLM call and that could maybe be like a conditional kind of like edge in some of these graphs. and and I think that's really good for places where you have more of these kind of like well-known processes and you care about kind of you care about determinism to some extent. and we still see people using Langraph for that and it's really popular in like financial services and financial industries for that reason.

16:07 at the same time I think a lot of what people were doing with those things were honestly things that were just more agentic. So, an example of this that we actually did early on was we we did we wrote a deep research example and it was a langraph application and it had like first you're going to plan and then you're going to get like these three bullet points and then you're going to fan out and then you're going to fan back in and then you might you might like review your work. and deep research is really just an agentic task.

16:32 Like I really think it is like there's so many different things you might go down that it's kind of tough to put that into like a graph-like workflow. And there's may and so basically we've seen this this trend towards more of just like yeah the these agent harnesses for doing things like deep research and then the way that you customize it starts to become skills tools prompts. We have a concept that we call middleware but that's con that's similar to like hooks in cloud code or something like that.

16:58 And and you can also add in I I think a lot of customizing it this way is about adding in deterministic things. So like adding in a check like goal mode. Hey, when you think you're done, run another check to like check whether you satisfied the goal. Like that's an explicit deterministic check that you kind of do and then if it fails, it goes back to the start. So this adding some determinism in there. And so great, you can do that with middleware, you can do that with hooks and things like that.

17:20 And so you that's how you kind of start to customize some of these harnesses. and but I do wonder you know talking about the difference between different domains like coding for a lot of coding tasks you need a full-on sandbox because you want to install arbitrary dependencies and you want to and you want to spin up a server or something like that. we we've seen some back and forth when thinking about kind of like document processing. How much of that do you need? Do you need kind of like just access to like a file? So we have a concept in deep agents of like a virtual file system where you can just interact with with a file system. You can read and write files using the same way that kind of like a coding agent would, but you don't have to spin up a sandbox which is like a heavyish piece of infrastructure. So it's a lot easier. And so we we have we have one customer that early on they were they were they were just doing that for their document analysis bit.

18:14 And I caught up with them the other week and they're like, you know, we we actually moved everything into a full-on sandbox because people want to use like PowerPoint CLIs and like yeah, you know, you can't really mock that. You kind of need a full-on sandbox for that. Versel has their just bash thing, which is another thing in this vein where it tries to like mock out the the bash file system. Pyantic has Monty, which is kind of like a fake kind of like Python interpreter. and yeah, so like I I I think there so on one hand, yeah, maybe for some of these other tasks, you don't need the full-blown coding stuff. Maybe you do and you and you kind of just do want like all these different CLIs installed and that's the way to do it. And so one of the things I'm thinking a lot about right now is like, yeah, what what like great skills, MCPs, instructions, middleware, absolutely those are how you customize it. Are there like deeper differences than that? And I I I I I don't know the right answer. I think we're still thinking through that.

19:10 >> Yeah, it's it's so true. And like even like installing a browser on a sandbox, like that's not very sufficient, right? There's so many extra steps you'll have to do to get like the browser working on the web. There's like these web bot off protocols to allow agents to go places or observability to record the browser session. Like how much of that are you scaffolding into the sandbox itself? How much you relying on third party providers for? I do think that there is some clear stack and in a way a browser is a sandbox, right? because it you know it's running its own you know JavaScript in the memory web assembly can be working there too.

19:40 >> Well so okay question about that because I think one thing that's popped up in like this sandbox usage is like how how do you like what's the right architecture for the agent and the sandbox? Does the agent run in the sandbox or does the agent like yeah to separate the brains and hands and run outside and connect and so I'm curious for like browser stuff like >> you mentioned you know in some ways a browser is like a sandbox. Do you see people mostly running this the browser whatever as a separate thing and having the agent over here as a separate thing and just talking to it as a tool or do you actually have to colllocate them?

20:10 >> Yeah, I I've seen more and more that people are separating the agent brain from the tool calls especially because you want to be able to like look at the data in between and make sure it's not sending any bad data towards the tool. Like let's say like let's remove like sandboxes and browser for a second just to keep it high level. Like let's say your agent has a tool to make an API request or something. It just has a network tool. Like you probably want some layer between the agent saying issue this network request that looks for like PII or social security numbers and strips it out before it goes out to the web. So like separating those and that like like control layer makes a lot of sense. Also to avoid like prompt injection coming back from the tool into the agent like having those separate is important. You kind of want have a safety zone around your agent logic. But then yeah, for the browser I think that maps to all tools like all tools probably should run outside the agent loop or sorry the agent harness in some sort of safe environment that could be destructible like one-time use. You probably don't want to have tools reused across the same sandbox or same like virtualization layer because then what if some pollution happens from the last tool that's called or how do you like manage the file systems across those tools? Like all these things are just super challenging for us to figure out and and in the end I'm always looking towards what are our customers doing.

21:18 It's trick because you hear customers speaking very different things and I think your customers are probably even more diverse than ours because like some of my customers I describe as like the cracked engineering teams that really push things forward and then other customers are really getting started on their AI journey and trying to build for both is very very challenging and you almost have to move up the abstraction layer from the crack teams to the more like newer teams or enterprisy teams.

21:42 How do you think about that? Like how are you shifting the abstraction layer across the two different types of customers you might have? I think we looked at we we we look at those AI build engineers for like where where the future of agents is going 100%. and and not everything, you know, not everything going on there is is is going to be kind of like enterprise grade or even like the right way to do things, but people are experimenting.

22:06 And I think that's really cool to see. And I think if you see people experimenting with the same thing in three slightly different ways, you're like, "Okay, there's something kind of like interesting there." And I think that's and so you know when people do bespoke things if it's the if it's the first time we see it and they're like hey like I'm trying to do this bespoke thing there's nothing easy and kind of deep agents to do that. I think that's where now we feel good about our open source story where there are different levels of abstraction and there's like langraph at the lowest level and so if you want to do anything you can do it in langraph like you can use langraph for just like arbitrary things like >> you fork it to like there's always an escape hatch into the lower levels of the >> exact so it's like langraph lang chain deep agents and depending on what you want to do you can kind of go down those those different layers and so but but then like once we see people doing like the same thing like yeah three times we're like okay you know like like an example of this would be offloading of tool calls. So tools return really large responses. I think it's generally standard practice now, but wasn't like a year ago to basically dump that to a file system and say, "Hey, here's the first like thousand characters of that, but if you want to read the rest, you can like go just navigate this file." And so now that's part of deep agents. And so we try to do that because absolutely there are a bunch of teams that want to take I so I actually think there's two things here. I I I think like it's yes like maybe they're not as AI pill as others, but I also think it's like there's all these lessons that we're learning from coding agents that are actually just more broad than just like you know a coding agent that you run locally. And so when a lot of these people want to take these best practices and use them for their first party agent which may not be a coding agent and may not be may not even run in a sandbox or things like that. But yeah, we we want to take these common things and bring them so that everyone regardless of whether they're building coding agents or not, regardless of whether they're AI build or not, can kind of like take them and use them to build agents of their own. So that's kind of how we think about balancing it.

23:59 >> Yeah. And it makes sense. I mean, I'm starting to think a lot more about where the models are going to go because I think like if we look back two years, things like things have come and gone quite a bit. remember one of the labs launched like this DAG builder this workflow thing and then like I was like everything's over and then I've never heard about it again right like computer use launched there was like this thing sucks it's never going to work like operator was not that good that actually held back my industry a little bit like it's computer use is bad like no it's changed right like we've had these highs and lows and these head fakes around technology changes that can make it really hard to predict what's going to stick and what not what's not. How do you think about that for these like bleeding edge innovations? Like how do you tell if something is like a long-term, you know, innovation or just a temporary thing?

24:45 I mean, I think so I I definitely hack around with a lot of stuff and so like whenever I'm trying to really understand like a product or technology, like I'll just go build with it or use it or try to build something in that way or something like that and then if I think it's like reasonable and and exciting for whatever reason, either it makes it easier or it enables something that wasn't before, that's how I get like the most conviction of things. I think it is like a weird mix of being realistic about where things are but also optimistic about the future. Like yeah, we had something in lingchain called agent exeutor in like November 2022 but like we did not go all in on that because we realized that you know it just didn't work at the time. And so like you know counterfactual what if we had what if we were just like hey we're just we're going to start building an agent harness in 2022.

25:40 >> I like I don't I don't know what would have happened like you know would like would people have used us? Probably not because you couldn't really build anything with that. So I think it's a weird mix of like what can you build now and what can you be practical with now and then yeah what can you architect for the future. My I think my general philosophy has been try to not like make any crazy wild kind like or what's a better way to say this. I think my general philosophy has been like build build up kind of like a a a team or a process or an ethos of just iterating super rapidly rather than like being super dogmatic and sticking to like yeah this is kind of like the way it has to be done. And obviously if you can make that bet on like, hey, this is like outlandishly true and and and correct bet and I'm going to stick to it for like 3 years that is like incredibly impressive and like you'll there's probably a lot of value to be captured there.

26:36 >> That's browser based, right? I mean like I I I do agree though. I think I made this dogmatic back on the browser in 2023 like a can use a browser and like there wasn't much evidence of that and like for the first I'd say almost a year browser automation done by a large language model was super hacky like we used to draw bounding boxes on stuff and like send it to screenshot and we'd annotate the screenshots for the BLM and so much has changed but I do think like those like having that core founder conviction in certain principles and ideas that's what makes our companies, you know, go really far and then at the same time you have to have some degree of skepticism about like what is the right way that fits onto the world like it's a worldview and then there's reality >> you have to like iterate on that as well like I mean when you were talking earlier about like yeah you know initially people wanted the higher level thing but now they want the lowle like that you know like the core idea of lang chain has always been that people want to build their own agents for their particular domains and their particular tasks what that has looked like has absolutely changed and so I think like there's like yeah having kind of like a mission and and a focus. And then there's there's more kind of like small micro things which actually matter a ton because if you get those wrong like yeah you can be like yeah you know I I built an agent harness in 2022 and and you know no one used it why not it's like yeah well that you know like directionally like interesting yeah people want to build agents but like the micro was like yeah that's not the right way to do it at the time and so I think there is this balance which is tough to >> yeah and my road map has become more and more dominant about like second order effects like you know for the browser stuff a lot of time I'm spending right now is on agent identity and how we the browsers lo in how make sure these are trusted and like browserbased in the future might be more of an identity company than a pure infrastructure company because that's a really really important problem to solve with a lot of counterparties >> for you and I think some of the stuff you did around observability like the second order effects around agents like you have to observe them and then you can use that data to actually make them better and self-improve and reflect back what are some of the second order effects that you've seen from like lang chain or agents built on lang chain and like how have you oriented your product around that I imagine some stuff on the data and the trajectory side has been very useful for your customers.

28:40 >> Yeah, I mean I think the the core of both the open source and the observability in Eval has always been basically yeah how like how do I build an agent to do a particular task and you have to custo you you have like this cognitive architecture in this model but you have to customize it in some way and so you have to have the knobs and the harness to customize it and you have to have the instrumentation with observability eval to measure how well you're customizing it and so that's what kind of like brings them all together.

29:05 the ways of customizing it have absolutely kind of like changed over time and I'm actually very excited about where they're going in particularly with a lot of like post-training and and RL like things. so primarily to date people have largely customized things by doing context engineering or harness engineering or prompts and scales >> looking at the evals running the thing back and forth type of stuff. >> Exactly. and and and and today they've they've stayed away from actually changing the weights of the mo like yes they'll swap models 100% but they won't change the weights of the underlying model. and that is a direction we absolutely find ourselves getting pulled in more and more as I think post-training techniques become more common as open source models catch up in in just the the base kind of like strength and as people develop these agents that are getting used more where they actually have this data and so there's a bunch of different techniques that largely either involve using the traces which great like Langmith helps kind of like curate those or using the eval again Langmith helps curate those and and so that's been a direction that we've absolutely found ourselves getting pulled in more and more.

30:11 >> Yeah, that's something I also didn't predict. We're spend this whole podcast talking about stuff that we didn't predict or has changed which is good. I mean because we I think you and I lived through a couple years ago now like I didn't expect how much open- source models would catch up and how much postraining would be real. I mean fine-tuning was a thing in the very early days and it kind of helped a little bit but it's nothing like it does now. I've been really shocked not just about how the infrastructure has changed to the needs, but also the model capabilities and the influx of different types of models for our customers. I'm wondering, you've been set up really well to help customers train models or postrain models. How do you think that they're using that right now? Do you see a lot more of your customers using their own models? Do you see a lot more of them saying frontier? How does it balance?

30:52 >> a lot of interest I think for people who have high volume. I think actually probably shockingly large number of those are kind of like using a post-trained model generally not to drive the core agentic loop but maybe for simpler things like classification or extraction or something like that. but that's that's still in in in the grand scheme of things a small number of customers that are actually doing it. And so yeah, one of the thing one of the things that I think at Langchain we've always tried to do is take kind of like some of the advanced topics whether it's you know building with LLMs, building agent harnesses and make them kind of like approachable and easy for folks.

31:29 And so yeah, we we we're spending a lot of time with you know the the more AI pill companies trying to figure out yeah like what what exactly are common things they're doing to to post train and wear like what types of problems is it actually the agent loop or is it the classification and I think right now we're seeing it's generally simpler things although they're experimenting with the more core agent loops and then we want to make that as easy as possible for for everyone. I remember I you mentioned kind of like fine-tuning was like kind of a thing and then went away and like I remember chatting with someone two-ish years ago and I was very skeptical on kind of like fine-tuning then and I think in hindsight that was probably right like I think open I deprecated their fine-tuning service or something like that and so there's another wave of it this time and I I don't know like what >> what do you think is different this time? Like I I mean I feel a little bit more bullish about it for some reason now. I I'm not sure if you feel the same way. Like what do you think is different about this like post-training era now?

32:24 >> Yeah, I think I think like the the core openweight model is just better, you know, like the stuff that you're able to post train on is better. I think we have better ideas like what types of data we need. There's more knowledge going on overall. And I mean inference is I think much more available. I don't know how much how many GPS we had floating around in like 2023. I think there's a moment there where it was like very very hard to get. But it seems like nowadays you can experiment for a lot lower costs and actually see some semblance of results.

32:50 The few, you know, fine-tuning things I did early early on were just like you need a lot of data to make something really good. It had be really clean data. I think now the error bars have just increased on what you can do. And that kind of leads into a topic I'm curious to talk to you about, which is like it feels like what everyone is optimized for has changed. Of course, everyone wants accuracy, but accuracy is almost becoming table stakes. I'm not going to say it completely is, but you know, it's it's really getting there.

33:13 And now I'm seeing these really interesting trade-offs on cost and speed. Like if I had to rank the three, I think people care about accuracy, cost, and speed. But more and more of our customers, especially those building general agents, they care a lot about latency of inference because latency of inference can be really really slow for some of these like frontier really large models. How have you seen your customers optimizing these ones? >> So we we ran a survey for the past two years, I think, where we asked like what are your biggest challenges or priorities? I forget the exact phrasing, but like what what do you think what's hard about building agents? And I think for the for the last two years, like quality or accuracy was basically double that of of speed and cost. and those were about the same. so I'm excited to run it again this year and I'll and I'll let you know what the results are. I I think I I do still think kind of like qual so so for like for general purpose agents I would bet like yeah co like cost I think for things where you're really trying to drill into a particular use case I do think quality still matters a lot I still think that's a blocker I also think that like as a as as models have gotten better maybe the existing use cases are now like saturated but we are a creative species and we come up with new things to throw these these models at and I think that that bar keeps on rising. So I actually do think kind of like accuracy and quality is still probably the biggest bottleneck for for a lot of folks. but I'm excited to run the survey. I wouldn't be shocked if it was if it was closer than before.

34:42 I definitely hear there's there's also different types of agents, right? Like for coding agents, I think 100% people right now are waking up to cost as as one of the main things now. I know internally we are like looking at this a lot and we think yeah for a lot of tasks quality is basically good enough based like use you you know choose your harness choose your model it's probably good enough for 50% of tasks I don't know what the right percentage is but for a chunk of tasks and so now we're thinking a lot about costs and I can tell you internally we spend a lot more time thinking about the cost of these models and the cost of these harnesses as opposed to like trying to measure oh is it 95% or 96% like >> yeah do it's domain specific quality is very much better coding is I think is really really solved and on the cost and latency side there's like a thought experiment I've been running about like okay let's say we solve quality let's say like cost becomes much more palatable or we can route appropriately to solve those two >> inference speed let's say that gets really really fast yeah I think that puts a lot of interesting pressure on the infrastructure itself like the agentic infrastructure I think right now a lot of infra companies can get away with you know a few hundred milliseconds difference like in the sandbox box wars.

35:53 I've seen battles over the P99 of startup time and it's like we're one millisecond faster like okay who cares but like as as models get really really fast and and cheap and accurate. those things are really going to matter a lot and I'm curious as another fellow infraounder there's not many of us how do you think about the needs of models you know the infrastructure serving the agents in the future? Yeah. Well, I I I actually think probably the bigger burden is like the infrastructure serving the tools that the agents access. So like sandboxes again, if you're running it as a tool, like I think yeah, you need to be able to query it and whether that means spinning it up or serving it from a warm pool or whatever, like you need to be able to query it fast, get a result back fast.

36:33 but but same with like every tool to access to access anything that you want these agents for. So an example of this that that we actually didn't do for this reason, but I think will benefit a lot. So So we have we have lang. It stores a bunch of traces and those traces can often be useful to give to coding agents or any agents for a variety of reasons including like reflecting on what other agents have done running experiments things like that. one of the things that we did recently is we rearchitected Langmith to be built on top of Smith DB which is a database that we built in house. My co-founder Ankush did all of it. So if you ask me any more info questions I'm going to I'm going to you know yield and and and ask him. But it's it's a lot faster at a lot at querying traces and querying feedback and attaching feedback and things like that. So now when we give it to agents, it's a lot faster for them to use these things. And I think that speed up like really matters. And again, we so we going back to something we talked about earlier actually like building for humans, building for agents. We did that for humans. We did that so we could populate the UI faster. But it actually turns out to matter I think a lot more for coding agents because again as as they get longer and longer running, they're just going to be hammering these APIs. And so these APIs which are just it's it's just a database, right? Like that's not like agent specific infra. And this is what I mean by like I think just infrastructure in general is going to be interesting.

37:45 Like if you're exposing a database to an agent, it's going to get hammered by a ton of queries. And so you got to think about how to basically scale for that. >> Yeah. It's almost like a multiplicative effect where like for every human there might be hundreds of agents and those hundreds of agents might be calling hundreds of tools that also might be reflecting on themselves. And this agent to agent world is going to be very interesting to see how it all plays out.

38:06 I think that the the protocols too allow people to call tools like MCP and the governance around agents is one of the second order effects I think a lot about. It seems like sure we may have at least a way to use tools call arbitrary services but provisioning them allowing them to act on behalf of this person that's been something that I've seeing a lot of activity around. How have you thought about that as part of your greater platform or strategy and and what are customers asking for?

38:30 >> Well, you had mentioned earlier that you guys are are kind of getting deep into identity. I would actually love to hear what you guys are doing there cuz because that's coming up for us now because like we're like people are building agents and like they want to give it exposure to tools and MCP's a protocol that is better now and people want to use and yeah there's this question of like how do the agents interact with these tools? Do they have their own credentials or do they act on behalf of users? And then when you start getting I I think it's actually like easy enough when you're kind of like in single player mode, but when you start getting into like multiplayer mode, what like imagine a shared Slack channel and you're chatting with an agent. Who who whose credentials do it use? Does like does it does it use my credentials? Does it use depending is it a fixed credentials like and there's actually different projects that tackle this in a different way. So, so Claude tag the way that they handle it as far as I understand is basically there's there's a set of credentials and when you interact with that agent in a channel no matter who you are it will use those type of credentials. and a bunch of other projects take on different approaches there and so I actually don't know what the best kind of like approach there is. and so we're starting to think about that. I'd be cur you mentioned it in the context of kind of like browsers. I'd be curious what you're seeing there and how that kind of like >> Yeah, you have the easy path. At least it's like protocols and APIs for this stuff, right? Like in our world like we're working with the web that's not AI first. Like the logged in web is where aentic identities be solved but there isn't a protocol that we can bolt on top of. There's like SSO and and SAML and like OOTH that kind of can help here but it seems like agents are going to try and access services on behalf of people.

40:02 And maybe once again this goes back to how you and I have different views here or different beliefs. Like I might believe that I think my agent should be a delegate of me because I probably have like a limited scope of access. And you can create a surface account for an agent if you want it to log in as its own self and have even less access. But I think for the majority of agents that we're going to see out on the web, they're going to act to the extent of which the owner has permissions. And the best way to limit this in like an enterprise context is like have them sign in with octa, right? Like the human talking to the agent has some degree of access in octa or their SSO platform and the agent probably needs to mirror that.

40:35 So I think that like working with the existing web and the existing credentiing systems and the existing arbback and SSO allows us to kind of speedrun a lot of these things because our part of the web the part of the world we touch doesn't have APIs and MCPs. So I'm mapping a lot of our primitives onto existing identity primitives that humans have and that's been kind of the way that we can kind of really be this compatibility layer over the existing web but add like an AI native connector for internet websites.

41:00 And have you found that in enterprises like everything that you'd want to give agents access to is behind Octa or is behind kind of like this this off sign up? >> Oh, certainly not. I mean, sometimes it's behind the jankiest website login you've ever seen that was built by somebody in 2006, right? Like there needs to be password sharing or how do we, you know, allow you to let your agent log in one time in like a plaid like experience then save the connection so that you can log in on behalf of them in the future. All of these things I think you look at them you're like man there has to be a better way and there will be a better way over time but it seems the demand for agents is is so so high and model capability in my opinion is getting there to solve most web-based tasks but there's this overhang of model capability where the models can go solve the task but they can't actually access the assistance they need to solve the task and that diffusion of model capability is where we think it identity can really play a big part and we're working a lot with partners like octa or one password to try and find ways case to securely share credentials with an agent one time so it doesn't actually go to the LLM or how do we configure SSO and OOTH to allow your agent to request access and you get a little push notification saying can I log in more and more of these patterns are popping up as computers gets good and there's been these several like fundamental moments in computer use where things have happened and people realized oh AI will use the computer and it's like brought up these discussions so I'm excited more and more as they become productive conversations and people use people view agents using a browser in the web as an inevitability and not just like a hack.

42:27 >> How mature do you think some of the identity things are here for for browser use in particular? Like are we making it up right now and there's no real standards? Do you feel like do you guys have a product here right now at this moment? >> We've been working on identity for about a year and have been announcing partnerships along the way. Like we did our first partnership with Cloudflare last year to talk about this thing called webbot off the way to say agents are allowed to browse to certain websites by identifying themselves. A few months later, we announced browserbased plus one password, a way for us to allow agents to actually connect their one password vault to our browsers and securely share credentials.

42:59 So, it's almost similar like we we've had these primitives for a long time, but I think people are now getting to the point where the models are actually running to these parts as opposed to hard coding everything up until or after the login step and then the agent takes over. So, I still think there's a ton of work to do and a lot of this comes down to the socialization of what these pyramids are and should be. Similar to Agentic Engineering, I felt like when we saw the claude code rush of winter 2025, a crazy time for me to take a vacation, I have to add, we're starting to see that more and more in different parts of our industry. And it does seem like if you can, you know, get in the discourse in the right way and and be a part of, you know, Twitter, I guess, where people are talking about this, a lot of these primitives take shape really, really quickly. But I'm a very pragmatic guy.

43:42 Trying to enable agents to do the task they want to do in the most simple way possible is really what I seek to do with our agentic identity products at browserbase. And there's a long road to go. The internet is a very large place. And there's probably not like a one-sizefits-all solution. You're just going to take down parts of the web piece by piece to figure out the best way to connect to it. >> I think you were kind of getting to this with like there's just these spikes that happen in kind of like popularity of projects where there's cloud code or open claw. What do you think like the project would be that would make this happen for kind of like web browsing?

44:14 >> I think it's already happened. I feel like Codeex computer use has really taken off and I see people tweeting about all the time. People are begging other people to use Codex computer use. And it's funny, everyone bought Mac minis for their house and they're like, "Yeah, we'll just run a little agent on it." And like that essentially is a device that allows you to tunnel traffic through your home IP address to handle some of the blockers you face when using the web. So when I saw people buying Mac minis, I was like, man, this is an infrastructure problem that needs to be solved because clearly the models have gotten good enough and that is the like expensive infrastructure solution. So I think it's been a series of like different announcements, different launches, different things that are happening and it feels like the consumer use case of agents often drives the B2B use cases. I just met with an enterprise customer today and the guy turned his iPad around. He's using instinct and I was like, "Oh, cool." You know, it's one of my customers and he we're talking about how he wants to make this work within his large bank and like do something similar internally. So, it's like I often look just like I look at like the AI build teams. I look at the consumer use cases of AI that a lot of people are using to look where enterprise is moving and where the B2B use cases are because often times the infrastructure will carry over similarly but with probably a little more governance and policy that's required. I know I don't know that much about Codex computer use. Is that running like on your local computer? They or do they do something like browser based and send it off to some remote company?

45:35 >> It runs on your local computer for the most part. I want to hear a little bit more about your journey as an AI build founder yourself cuz lane chain is pretty big now, right? >> Yeah, we're like 360 people. >> How do you do that? I mean like what is it like? Are do you have agents managing lane chain? Like I want to hear about your AI used to run lane chain itself. >> We have agents in some place. I I actually think when I'm talking to candidates and and all they can talk about is using agents to run and do their job. I actually think that's a bit of a negative signal. I think like I think there still needs to be a bunch of taste in terms of how you apply agents and where you use agents. But we do use agents in some place in terms of where we use it. I actually think we use it most in kind of like the go to market or So, we have a pretty we have a pretty good kind of like go to market agent that does qualifying of inbound leads and helps with outbound as well and that's built in-house and we actually have two to three kind of like applied AI engineers that are building that out full-time. along with along with a marketing leader who works basically just just on this and that's now extended into it started off mostly in sales and now we now we have kind of like I forget the name of it. I think we call like content something but we use that to generate all of like our GIFs and images and things like content studio is what we call it and so that's like an in-house agent all exposed just in slack. So Slack is I think we technically have a web app but Slack is the primary interface and then we use coding agents a bunch. so we we have our own we have our own version of of basically a cloud-based agent. We call it open suite. It's actually open source if people want to go check it out. But we again expose that in Slack and then and and use that pretty heavily. So that that's actually how I put up most of my PRs these days because I don't have a ton of time but I see something wrong and and then slack at something.

47:13 and then and then I have basically a executive assistant kind of like agent that's basically going over my email. So triaging it, pinging me when there's things that need my attention, trying to draft responses, finding calendar times, things like that. So that's what I've done personally and that's what we've done in the org. >> Yeah, I found that myself. I have my own agents I use for email, etc. but maintaining your own agent can be a lot of work. How much time do you spend building the machine that helps you execute as founder of Lang Chain or did you just oneshot it and all of your tools are working pretty well all the time? So, you're telling me that when someone interviews for Lang Chain and they say they use agents for everything, it's a red flag >> a little bit.

47:55 >> Why is that? Tell me a little bit more. >> I think there's things that agents are good at and there's things that they're bad at. And I think like and I think the work that you produce, whether it's you've done it manually or you've had agents do it, is your responsibility. And so if it comes out looking like crap, then like yeah, you're on the hook and I don't care if you've used the agent to create something that's bad.

48:13 Like you you've used a tool in an incorrect manner. Like why should I care about that? And so I think like I I think like the focus on tools more than outcomes is is not a good thing. I think when I think when people start talking about the outcome they want and then kind of like get into like, you know, specific places where it could make sense, I think I think I like that answer. I think when I ask a question, it's like, "Yeah, I'd use an agent for that right off the bat." or like trying to force an agent into something. I think that's been We've had to deal with a ton of AI slop recently. I don't know if you've seen the same.

48:42 >> How are you fighting slop? I have. I'm curious what you're doing internally. >> It's a good I don't think we have a perfect answer for this. I think things that we've done are we've had our VP of engineering basically talk about AI slop. Talk about Clay wrote a really good blog on their four points for kind of like AI writing. >> AI writing. Yeah. >> Yeah. And so basically talk about that. I think we added a slack emoji that's AI slop. I've been using panggram more when I see things. Again, if you write something good, I'm not going to care about it. I'm going to like you know, post it or or send it or something like that. If you write something bad and I can tell that it's AI, I'm going to go to Pangram, post it in, it's going to say it's 100% AI and then be like, why are you doing this? This is bad. And so, I think like I think you kind of have to like call people out. Like I think like Yeah, I think we're seeing and and I don't think we're unique in this. I think a ton of people are dealing with like there's AI slop all over base. Yeah. And you know, I don't know if this is weird or hypocritical to say as someone who helps people build agents, but it yeah, it annoys me. It pisses me off a lot to see all this AIS.

49:43 I think it's really not like well thought through. >> Yeah, it's funny. We're we're building the infrastructure on an everchanging world of different models and problems to solve, but we're also simultaneously building companies on an everchanging world of like how companies operate and what's going on and if you will use slop or not or how it changes. So it feels like these two worlds where we're constantly seeing changing environments and having to really challenge them. it's pretty fun. You know, we get to live in this great time. How how do you imagine Lang Chain develops over the next few years? Like where where is it going from here? What are the unsolved problems that you want to take on next that I can check in on next time I do this podcast?

50:17 >> I think the big thing we're thinking about now is like what is the appropriate like runtime for a lot of these agents. So I guess kind of to your point earlier around quality being like a little bit again I still think it's a concern but like other factors coming into play. like I think we're we're going beyond observability and evals and we're doing a bunch around deployments and sandboxes and gateway and really this idea of like a governed runtime when you want to run an agent that's not on your laptop and you want to run it at scale for a bunch of end users with proper permissioning and proper guardrails and proper cross controls and and and have it scale up and not fall over. Like what are all the things you need there? And I think we know some of those components. I don't think we know other components but I think sandboxes durable execution model gateway some form of like off gateway and that might look different for like whether you're using it in like an MCP or as I learned today as in browsers.

51:11 but off super important. and so just thinking about how we can either build or integrate with all the best things there and provide kind of like because I think one things that's happening is like there's all these pieces that are now coming into play. And so like deploying this at scale is actually becoming trickier and trickier. You need to assemble all these little Lego pieces. And so one of the things we launched that I'm spending a lot of time on personally is manage deep agents. So manage deep agents basically binds the harness with the infrastructure. And the reason I'm I'm so excited about that is I do just think there are all these pieces of infrastructure and having to assemble them one by one is like really time consuming and we don't want people to do that. We want them to get up and running with agents easily in production.

51:49 >> So once we build all of that it's mission accomplished and we're done, right? Agent engineering is solved. >> No, there'll be more things. I don't know what but there'll be more things. >> Well Harrison, I'm confident that you're keep figuring them out. It's been really fun having you in this conversation today and thank you for taking the time, man. >> I've enjoyed it a lot. Thank you.