Code Mode: Why AI Agents Are Ditching Tool Calls for Code

0:00 My view of software in the future is it is all code mode. This principle used to be gatekeep only developers. Now you could only contribute in this way of extending software if you're a developer. You can now extend every piece of software ever and everyone can extend every piece of software ever. How do you make code mode so easily accessible where it's both like you said running untrusted code but in a way that meets the bounds of your application? Hey everyone, today's episode is going to be completely about code mode. And what code mode is is something that you're probably likely familiar with, which is if you've ever seen an agent use the bash tool, that's the most basic form of code mode. So we talk about how that works, how you generate code, and then how you run code. We talk about all sorts of analogies to this in previous software engineering worlds like like SQL representations, step functions versus code. We talk about when you should use code mode. And most interestingly, we had two sections of today's talk where we talked about Kyle's actual implementation in the human layer codebase for how they build code mode to do dynamic dynamic searching throughout their or dynamic modification of their CRDT data that they're holding. And then secondarily, what we got to get a little sneak peek of is how you actually do code mode in possibly BAML once you have both type safety and the compiler baked in into the same runtime. And that comes with security trade-offs, that comes with that comes with all sorts of open questions that we kind of go through. So, I hope this is interesting and let's get started. I'm Vibbof and this is the AI that works show. And as per usual, we like to go and show real working systems that you leverage AI into working pipelines. Today with me, I have Kyle, who is Dextra's co-founder, but I'll let him briefly introduce himself.

1:44 >> Yeah. Hey guys, I'm Kyle. I'm co-founder of Human Layer. we work on really cool coding agent stuff. >> And I think today's episode is something that I was actually really excited about for quite some time. And specifically, the thing that I was excited about is talking about code mode. Now, I think that's a word that either is not described very well or a word that is perhaps described in totally different ways in many and depending on who's using it. So, we should start off really quickly, Kyle, with how do you think of code mode? What is it to you? What does it mean? What does it enable?

2:20 >> It's a a way to give an agent more tools or to let it compose tools differently. When you think about how an agent normally calls tools, you know, it'll generate one or multiple tool calls in an array, right? You execute the tool calls, you submit the results to the API and it roundtrips it, right? It does more inference, it processes that, it can call more tools if it wants. but it has to do that in order to get the output. Code mode is a way of like giving the agent almost like a runtime where you give it kind of tool definitions or stubs and it can write code to use those tools to chain them together to pipe them to process you know filter map reduce whatever them so that you can execute a longer or more complicated sequence of actions than you might you know otherwise be able to do with just normal tool calling. There's a lot of different reasons why you might want to do this. but that's that's kind of like how it works at a high level.

3:19 >> So, if I were to break down this code mode box, what is the main thing? It sounds like it's a thing that's leveraged by an agent. So, an agent kind of has access to this somehow. >> Yeah. >> And it produces some like can I call it like dynamic output? >> Yeah. I mean it it gets you know the agent writes tokens out they get executed as code and then the output gets shown to the agent. Right. The the most trivial example of code mode that I think everyone that is watching this is familiar with is actually the bash tool right the agent writes code i.e. bash. It has a bunch of tools baked into that like runtime and the bash shell, right? That's just your Linux, Unix, Mac OS utilities, right? GP, read, cat, pipe, whatever. It's, you know, small sharp tools that the agent can like compose and do a bunch of like things in sequence instead of like having to say read this file and then get the result and round trip the API and use like a read and a GP and a glob tool. So like the cloud code harness for example used to have a bunch of different tools for like list files, grabb files, glob files and a lot of the cloud code harness is moving in this direction and other harnesses like pi have kind of always done this is just that the agent already knows how to do all these things just give it the bash tool because it lets it compose it in a better way. So yeah, bash is basically like the most simple form of code mode, but why do you think it's starting to get a lot of conversation happening now about what code mode actually is? Like where does that happen from?

4:48 >> There's a couple different reasons you would do it. it solves some some interesting problems. And the first is it one of I I think maybe what pe what got people initially excited about it actually was like code mode over MCPS, right? people had a lot of different MCP servers that they wanted to plug into their agent. And so by default, all these had to be added as normal tools into the agents like context window with descriptions and and like it would go list all the tools and if you had 60 tools, now there's 60 tools in your context window. And so what people first started to do was to say, "Okay, instead of like adding these as tools to the agent, we're going to like use a little like JavaScript or Python SDK to turn these into just like functions." And the agent can just write Python or JavaScript code to call these functions and compose them and to discover them so that you can have dynamic tool discovery and you can make them more composable and do like kind of multihop tool calling without roundtpping inference. So it it's more token efficient in terms of what starts in your context window and also more token efficient in terms of like what gets dumped out back into your context window if you need like multi-step like sequences.

6:02 >> Interesting. I think the more interesting part about this is if you had to system design code mode out and you had to actually go and decide like if your app should or should not use code mode. How do you go about that in your mind? >> It it there's a lot of different like reasons people use code mode. And by the way, for everyone else, like why I'm asking Kai why I'm asking Kyle this, Kyle has worked on code mode. He's added code mode features. He's done this. So there's a lot of nuances to this discussion that I think a lot of people like overlook in like a highle concept. So I'm just like really curious on like his specific engineering take.

6:37 >> Yeah. Like code mode is like one of those things where there's like a bunch of different ways to use it and like two people can be having a conversation about code mode and talking about like completely different things, right? There's like the MCP use case that I mentioned. but there's also like all kinds of other ways to use it, right? The codeex harness now supports code mode natively, right? and so like there's a bunch of different reasons you might have it. Like the default is not like, oh, you should have code mode because code mode is not super in distribution for the models yet. My take is like there's certain uniquely shaped problems that you should use it for.

7:16 and one of them is, like I said, if you need a bunch of different tools in your context window that you just can't afford to stuff in there and you need dynamic tool discovery, right? So, dynamic tool discovery and like lots of tools and needing to compose those well is is like one of the first things that would maybe make you think, yeah, I should have code mode >> and like you need dynamic tool discovery because like way too many tools.

7:40 >> Yeah, way too many tools. Like dozens, right? like an everything agent for example like if you think about like manis >> the idea that dozens is way too many is kind of fascinating to me but cuz in what other world >> in what other world of software do we have dozens being like oh yeah that's too much >> yeah >> did I spelled dozens right I hope so >> I think you did yeah >> okay >> so like for example if you think about an everything agent that needs to be able to check your Slack and needs like 12 different tools for Slack like list channels, read a channel, read a thread, send a message, send a DM, like search through chat history. and then also it needs to be able to do like all of those things also for your GitHub and also for your email and also for like six different social media and SMS platforms. like you get into a question of like there's good SDKs for all of these and so do you just let the agent use the SDKs which are kind of already like in distribution and you just let it execute it in like a JavaScript or Python runtime and inject some credentials into it or do you like go and figure out how do I like design a harness that like lets the agent decide which provider wants to talk to and what it's going to do in that provider and manage interop so you have like one like send message tool with a bunch of different channels or you just add like a tool for each action in each channel.

9:07 It's like a weird like kind of strangely shaped harness design problem. >> So I know so given I mean I tease this a little bit but you guys built code mode. So what what do you guys built? Show me what you built. Do you have a system designed for or do you want to screen share and show it live? >> I can try to pull it up. but the the thing that we built code mode for, and this is actually another like really good example of when it's it's very obviously the case you need to do this, is we were trying to have agents edit CRDT documents without getting like super in the weeds on CRDT. You have a data structure that represents some data. It could be a string. In our case, it was an XML document that represented like a rich text document. So, think Google Docs. And you want the agent to be able to edit it. and that data structure is being replicated across a network in lifetime. But the it does like the semantics of like an a read tool or an edit tool like the harness normally sees like during RL don't really make sense for that because it's a data structure and it's like the it's kind of like if you took like a link list or a tree in C++ and you're like well how do I get an agent to like edit this? The answer isn't like well I give it a read tool and have it like get a node ID and then like you have to have a super complicated tool interface that says like how am I traversing this data structure and selecting the node that I want to read or edit right it's like you just need to have it write code to traverse and modify the data structure because it already it can do that already but otherwise you end up with like a JSON tool with like 40 different input properties that's just like >> super out of distribution >> it's like the These two are basically the same. And we're like, and you just have this be like null to signify all or something. And this is basically kind of saying the same kind of stuff. But >> right, >> if I were to go and have the agent write something, I know what it's going to prefer writing this because what you're inventing is a shitty DSL on top of something that's well described.

11:17 >> It's like or like something distribution too, right? It's like the agent already knows SQL. Otherwise, you have to teach it what your little DSL means. >> Let me show you something that's so funny. do you know what step functions are? >> Yeah, >> it's like the exact same concept here. It's like you can have a I if I can pull an example of it and make it so clear. I actually what I should do is just ask chat give me an AWS step function example.

11:47 And I think this is probably the closest concept to like what code mode really is for many people is if you've ever used as step function or something like that like this is one way to write code where you literally write this. You add a comment you write start then you say these are all the states I have. This is how you go from one state to the next >> and this is kind of describing >> state chart.

12:07 >> Yeah, exactly. And then you can you can ask you like this is exact same equivalent code. It actually described it to me that way cuz like if you're trying to understand step functions and you're trying to be like okay well >> you can write this and I'll bring this to the Excal so we have it. You can either write like option number one of like the TypeScript code or you can write this. >> Yeah. at the top and like I think we all know which one it's not about which one I prefer like I can imagine which one is less prone to mistakes where can I take advantage of all the tooling benefits and stuff that we have like it's just so much more likely that an agent will get this thing right as opposed to the thing on the left right >> yeah right >> yeah like JSON is like weirdly like it's like very in distribution for the models but like they don't like tools that have like super large complicated JSON interfaces. They're just like not for whatever reason.

13:10 >> Well, I don't think it's a model thing. I just think a super complicated JSON distribution is just bad for humans, too. >> Like you're writing a serialization effectively. It's a serialization layer and JSON is amazing for serialization. It's not really good for like maintaining schema specs over time. You've seen like open API for example. Yeah, >> open API is another shitty example of like JSON trying to represent the world and like would you rather write open API specs or write pure TypeScript code or any language that you're used to writing? And I think the answer is often very clear.

13:43 >> Yeah. >> cool. Do you want to show off your code mode stuff? >> Yeah, I found I found I found the PR that >> I'll stop sharing. Let me take it over. >> Let's see if I can get this going. What we were trying to accomplish is editing a YJS CRDT. for like a rich text document which it models as an XML fragment which for anyone who's not familiar like HTML document nested tags tags can have content tags can have attributes but this is the API for like messing with a YJS XML fragment right it's it's quite complicated these are all like rich types and they're objects and by the way you also have XML elements which have their own APIs and then you have the text objects within them. And so it's it's like you you look at something like that and you think, well, okay, how am I going to like design a harness that like the agent gets to let it do all of these things? and like it's not out of the realm of possibility, but it's also the case that Y.JS is the most popular JavaScript CRDT library out there. And so LLMs are very good at JavaScript and at YJS.

15:01 >> Yeah. And and so like that's kind of the first thing that like popped into my head is like well the agent can just write the code to do this. And the other thing that you get into is to perform operations on a like a YJS CRDT, you normally need like multiple different operations, right? You have to get a node, you have to validate in the document, hey, this is a valid node. you have to kind of traverse the tree and then say, "Okay, I have the element I want. I have a reference to it and now I'm going to insert text here." And so like in JavaScripts, that's probably like a for loop and something to update a node, right? And so for the agent to do that, you would just have to have the agent call tools in a loop until it gets to the thing you want and then update it. And by the way, because that took 10 like round trips of the inference API, that might be a materially different document now because other people might be editing it editing it than it was when the agent decided on the edit it wants to make. Right? So that was the other thing to think about is you need to be able to call a bunch of tools in sequence and have that kind of execute at the speed of code so that the agent can just generate one string and have it be applied rather than like roundtpping GPT 5.6 six all x high API and like 5 minutes later maybe you have the update that you want and the document is entirely different and so it gets discarded. Anyways, >> I want to highlight that by the way like the key point that you said here for me was I want to execute tools at the speed of code. There's no freaking way you can execute tools at the speed of code if you're using an LLM to decide what tools to call. You just cannot because like you're going to hit network latencies of at least 30 milliseconds just like pure network bandwidth at a minimum town not even if token speed was zero time and then you have token speed on top of that.

16:50 >> So like you're talking about 3 400 millisecond fastest response time. There's no code in the world that runs that slow for dispatch loops. >> Yeah. There's not a single language in the world that runs for loops in two in 30 millisecond time frames. >> I mean, there might be, but it would be imp you would need to do real engineering to make it that slow is the way I would describe it. >> Yeah. Yeah. And like the other thing too is like you need a smart model to do this, right? It's like you can't just use a super fast model like >> you know like Luna or something. we're doing complex operations across a a a data structure that is quite large and that the agent needs to be able to like read and inspect and so you do actually need a reasonably smart model >> and so as the you you it might even take you more than one try to do this >> thing works.

17:46 >> Yeah. So this so >> this this does work. I don't have my like local devstack set up for it but like this is the thing that we built which works and I actually I I got invited to give a a talk about it for the online track at AI engineer worlds fair and I ended up declining because I got the the inerson slot but like this is this is a a practical solution to the problem and like we can talk a little bit how a little bit about how it works because like there's a bunch of other like there's a whole like dimension of how you actually design tool calling for agents based on like what type of agent you're working on.

18:23 >> can you go to the Excaladraw for example really quickly and perhaps like draw for me draw for me the loop that you're building here for your code mode system like what does your code mode system look like? >> Yeah. So the first thing that you have to like think about is you know the agent is basically like it gets a prompt and it like gets some information about like in the system prompt why don't we do this why don't we do our little context stack here right we have like Okay.

19:05 >> Right. And so then like we might have, you know, user messages that are telling it like what to do, right? And then the agent decides it needs to go ahead and actually call the tool, right? So it's going to say like execute code code string. >> Yep. and it knows that it can do this because the tool definition is in the system prompt. And in our example, there's a bunch of instructions about like what kind of how the like little SDK that we inject into the runtime actually works. So, let me see if I can find that >> as a TS script.

19:57 Here's we have shad cn imported or like e etc. Right? Like you can basically say like we have yjs imported >> mhm as >> yjs >> right variable. But so there's an additional kind of nuance here which is like so the agent has to execute this code in like our little runtime but it can't actually write YJS because the the YJS it to be able to do this it would need to be able to execute code against the YJS object. and the object like lives in our backend right this is this is for a cloud agent. So it's an entirely separate thing if you're doing a coding agent where you can just give it a little JavaScript or Python runtime and fork the process. If you're building an agent in like a cloud agent for a multi-tenant backend, you can't trust the code that the agent is writing and you can't so you can't execute it against things that live in memory inside of your backend. So what you actually need is you have to take you have to basically build a little SDK for it. and we're going to call those like bindings basically.

21:12 And these are methods that get injected into your like in our case we used quick.js >> quick.js runtime >> super lightweight like C++ or Cbased JavaScript runtime and it allows you to inject bindings into the runtime. So I'm going to go and find where that is in the diff. >> so like for example it hardcodes like what user ID that you're calling to. So the user it does this runtime doesn't even get access to like every single tenant. It can only query for that specific user or that specific session.

21:48 >> Right? So it that's too much. yeah we basically define a bunch of function interfaces that it's allowed to execute and like this is the code that just injects those functions into the runtime. we also injected like a console API so the agent can execute code and inspect something in the middle of that execution code because if it fails, we don't just want the agent to get an error message. It needs to be able to like see where the program state is at various points in it. And so that would include whatever it logged to the output. but so you have to design these little functions that define like what is the interface oops that the agent can actually do. So we would have you might have something like list nodes you know node ID or list child nodes and then you could have you know like >> so you're basically building a wrapper on top of YJS effectively >> right but you still want it to look as much like YJS as possible >> but it doesn't allow agent to access the JavaScript.

23:00 >> I'm going to push back. I'm going to push back here really fast because the thing I'm curious about here is the thing that inspired code mode and it seems like that wasn't the exact thing. There's a different reason for why it's really useful for you guys was that stuff is already in the training data. >> Yeah. >> Which doesn't seem like that's what it is cuz this like your nodes are not in the training data. They're close but they're not in the training data by default because you're inventing this API surface.

23:24 >> Yeah. But it sounds like the real value prop for you is you don't have to do a multi-tool call loop. >> You get to do a single tool call execution loop. So you get much faster execution time. >> So it sounds like actually for you it's actually not the fact that is that a is that the right is that the correct interpretation here? >> it it's yeah it's mostly correct. I think that is one of the main value propositions is the agent can write 40 procedure or 40 method calls to touch a JavaScript object >> without like roundt tripping the inference API. So that's the main one, right? It is also the case though that YJS is very in distribution and the little binding SDK that we wrote that gets injected into the runtime is shaped almost exactly like YJS >> and you tell >> and we gave the agent instructions on how to use it, right? It has a system prompt block that says by the way here's the SDK that's injected into the runtime and here's how to use it.

24:21 >> Okay. I see. So you're like here's the SDK. It's pretty much YJS. >> Yeah. >> Etc. Right. And you just tell it this. >> Exactly. >> That's cool. That's cool. >> With like But there's like a little more detail of here's the methods and the signatures. But yeah. >> So I'm going to ask a couple more questions. do you do compiler checks or do you just run it and wait for it to mess up? >> so we did JavaScript not TypeScript and so it's it's not compiled. just because that's like you would need to actually have Yeah, you would need to run a TypeScript compiler in your back end basically.

25:01 >> Yeah. And that's hard. >> right. >> Does this require a backend or is this running in the web server directly or do you have a separate microser here? >> Yeah. So this code was not super opinionated about it. It was designed as a module. You could you could deploy it inside of a service either way. But yeah, if if you wanted a TypeScript compiler, you would need to think about, hey, compiling TypeScript code can be expensive. TSGO is not cheap. and like how do you slot a TypeScript compiler into your back end like to be used >> need like Yeah. It's kind of complicated.

25:35 >> Exactly. Yeah. And and so like what you do instead is you just Yeah. And so you what you need instead is like when it fails you have to give it the error message and you give it like a console.log method. you redirect console.log to point to the your own capture log. Then you point it back to the loop and now your agent tool call agent loop has that access. >> Right. So then it'll like when the tool call actually runs it'll say cool here's the result.

26:03 >> Yeah. >> And error. And then here are the console lines. >> Yeah. Exactly. And you kind of get kind of everything for free in that way. >> I have like two quick questions really fast. How does quick how does quickjs runtime >> deal with this? Is it just like an in-memory JavaScript runtime? >> yeah, so it's like it's it's C or C++ based and so it's like it can be compiled to WOM and like deployed in a a backend that way. that's cool.

26:34 >> Which is why you need bindings, right? because you can't pass a JavaScript object like a CRDT, run it in your backend across like a web assembly boundary into like a like a sandbox thing that just knows how to execute JavaScript syntax but is not actually like part of your JavaScript environment. You could do something maybe a little more clever with like V8 isolates and and stuff like that. but like this is the kind of more more secure way to do this. Yeah.

27:06 >> I'm going to give a quick little shout out to if you haven't tried the Asian OS team. they do some interesting work in this kind of stuff too. might be worth for other folks check it out. No no no real obligation but you know they do some hardcore engineering. Okay. So I think this is interesting. This is like a really practical way to do code mode. It sounds like there's a real like if I if someone were to take the transcript from here, they could probably feed this to their agent and find their own application of this pretty easily because this concept of having bindings, having a runtime, remapping, console outlog that makes >> I'm also reasonably sure that this repository is open source. We use it for a bunch of things.

27:43 >> Agent layer is not open source. Yeah, I mean it's fine regardless of the fact like people can definitely like point to it and I suspect an agent can figure out most of this stuff with just the conversation we have. If you're gonna open source it, then yeah, just we can share the link to people for exactly what that PR is as well. >> Yeah. Well, I can also just make it public right now. >> Exactly. There we go. We just made it public live. then boom, done.

28:05 >> Yeah. So, I I do have a Go ahead. >> I do want to back up though for a second because there's a couple other reasons that you would want to do code mode this way. And there's a couple different ways you could do it. The Open Code team, for example, built code mode into Open Code and they just wrote their own little TypeScript interpreter. because you don't need processing because it's not a multi-tenant system. It's running on a single user box.

28:27 >> But in a multi-tenant system, it has to be completely isolated because this code is not trusted. And even if you limit which objects it's allowed to touch in JavaScript, if you can touch the prototype of an object, you can more or less execute arbitrary code. This is behind all kinds of different remote code execution vulnerabilities you've heard of. I think maybe like one of the React or like Nex.js JS vulnerabilities was like a prototype pollution thing in JavaScript, but generally like letting untrusted code touch an object prototype is incredibly evil. So you have to you have to sandbox it.

29:02 >> Okay. I actually want your thoughts on some stuff. So I think this is a really practical example. I want to talk about some non-practical stuff now. >> Yeah. >> Cuz that's what I like to really push, right? Cuz like practicality is fun, but why don't we take it to the next level? So I'm going to I'm going to draw some >> dude. Do you want to take the screen share of? >> Sure, I'll take it. And I think particularly what I'm really interested in is like here's where I find code mode most interesting. My view of software in the future is it is all code mode. And what do I really mean by code mode? I want you to think about the most useful kinds of software that everyone in the world uses. It's browsers. It's apps on your phone. It's extensions and VS Code.

29:40 What has made all of these extremely successful? It's basically being able to run code that matches some interface that extends that application to every niche user's use case. Like when we built BAML, we didn't have to go and update VS Code. We shipped the VS Code extension and that made it usable by every other user. When you want to make a new iPhone app, you don't ship new hardware. You ship the app that matches the platform that you ship to. And I think this principle used to be gatekeep only developers. Now you could only contribute in this way of extending software if you were a developer. And I think the beauty of soft AI is like this is really the untapped beauty. You can now extend every piece of software ever and everyone can extend every piece of software ever. It's and if you want to go do that, the question I ask is how do you make code mode so easily accessible where it's both like you said running untrusted code but in a way that meets the bounds of your application and that's what I think the really really tricky part of this system is because why why does not every system build like extension apps to itself it's because like you have to maintain an API contract you have to secure it and make sure it's not going to do something really dangerous.

30:53 And then you also at the same time have to understand the cost per brim of it. We just started talking about TypeScript and we realized and one of the things you were like I need a back end now to type check and like do I have to spin up a server just to type check the code. >> Yeah. >> Like that's that's real heavy compute usage and not just that that's like a real infrastructure bill you have to pay in a way that's like complexity that you may not want to take on. Clearly in your case wasn't worth it. So, we've been thinking about this a lot and we have some weird weird implementation of code mode that we've specifically been thinking about. Let's see if I can pull it up. So, this is and I I want you to take on this and I will show you the first example of this >> and see what you think.

31:35 >> All right, let's see it. >> So, the first thing that we're doing here is code is passed in as a string. >> So, this is the very very first example. Code is passed in as a string. You can go write this and I'm going to pass in some more code as an additional string that's going to fill in this string. >> So when I go run this code run what is this function called? >> Okay. put on run this. I'll just run render with plugin.

32:07 Demo run render plugin. What I want to be able to run is let's say the name is Kyle and the code I want to run on it is actually going to be something like name two uppercase and this should run widget Kyle because this just runs a name and name becomes whatever expression I pass into here. So if I did name dot dotsplit split on like empty braces join on like space and then did this you can kind of expect what it would do. So you're kind of watching me dynamically change the code on the fly and execute it as it's happening.

32:44 >> Yeah. >> But what hap what happens if I find a function that doesn't match >> and I'm looking for the foo function now instead of the render function. Well now you get render not found. >> Right? But what's really >> So you do actually have the type checking there. >> Yeah. So and lookup. But if I do something else and let's say instead of this I just did dotsplit. Now name.split will create an array type. It creates an array of strings instead of a single string.

33:17 >> And widgets name is defined to be just a string. >> So let's see what happens. you actually get a compiler error that says and you can render this prettily. We don't right now. where'd it go? Mismatch type for agents not humans. >> Yeah. Yeah. I mean this just needs to be printed out correctly for like a human diagnostic. But this basically gives you the same compiler which is expected string found string array. So it actually >> so you're you're giving it a string and your code is telling it to split this string and then that string is being passed into >> if I had written name.plit here and this will obviously not type check because name has to be a string.

34:05 This is a string array or more obviously if I did name.length length and this is what I plugged out. now it's not being passed in anywhere but I will get like expected string found int. >> So it's actually running the type checker to make sure that this code is one safe to pass. >> So it's like it's assembling the code and then doing type checking before execution. >> Exactly. So it's assembling the code and then running type checking before execution.

34:39 But it's doing another thing before execution. I'm going to change this string to return it. I'm going to say that render is a function that returns an intakes in a string returns an int. Now if I do this we all know render here is a function that takes in a string returns a string. So it's a slightly different kind of thing. So this all type checks everything is good. So we'll just switch this back to a simpler version of like two uppercase.

35:10 So what you're able to do now is you get a different error which is oops did that mess up code. oh that's funny. Yeah, there you go. So you kind of get this here where like ren the function render has a type of string string which is not a subtype of what you're looking for which is string to int. So you're looking for a function at string to int but it's it does compile.

35:41 It does work but it's not what you're what you're actually trying to get. It's like a type error effectively. >> So you're actually getting two levels of errors under the hood as you're trying to go execute this. So you kind of get the code mode on a very extendable plane if that kind of makes sense. >> Yeah. Yeah. And this is the other thing that's like really hard about building code mode too is like making the errors be not nonsensical or sensical I guess to your own right. It's like okay something goes wrong and it's a bunch of JavaScript code that's executing and like some of the SDKs or bindings might not have like obvious errors. for example, in Quick.js, you know, like the errors can be like super weird and it's hard for the agent to figure out what's going on. And it's actually like an entire like design space of how you do your code mode of like how do you design the errors for the agent to be able to figure out what went wrong and correct it. That's actually really hard.

36:42 >> It really is. I was like, what I should really be doing here is this bamlio.print.n in and I just print out the compilation error somehow. Oops, compile failed. So like if I'm running all of this through, this is kind of what I want to get. Oops, compile failed. Render has typed this, which is not a subtype of this. And you can kind of see what's going on now >> in a really easy way. And this is what I think a lot of people need to think about a lot more which is as you're executing down this train how do you make it so that the code you're writing is going to abide by contracts that you're defining at compile time cuz if you can only have defined that and that's is kind of what your bindings are doing I think where you're defining bindings which are like your compile time content and you're describing it in some other way. So, like if I take it to like the nth degree, let's see if this thing is working.

37:41 You kind of want something like this. Let's see if this one works. actually, let's go back to another one where like if you have a contract that described in code as in summary is a field that is like summary and key points is a field that is like a string array. >> Uhhuh. and the agent generates this code. You want to be able to say something along the lines of, hey, I have I have this code that the agent generated. Summary maps to summary. Key points map to key points. Now, on this example, can you run this code? Well, I can run this code and produce a summary.

38:22 And I can say that these are specific things that are only for this type. So, you're kind of extending the type. So, like if you have like a broken point right here, the agent generated this code. Notice we don't have key points anymore. Well, before we even install the plugin or extend it, we get told, hey, we're missing key points. >> Mhm. >> At compile time, I've required key points. You haven't done that. So, let's tie this to bullets now. And now this works. And now I can execute this again.

38:53 And this actually renders an LLM call. And it shows me the other stuff as well. So you're kind of able to extend software behind the scenes without even having to go do this. So if you've totally broken everything and it like for example key points is now a string instead of a string array you get told boom it's require string array but class field is only a string. So you're kind of again able to extend the code without having to do much behind the scenes both on like an interface layer where you can define the interface and also on on the what's it called?

39:29 excuse me and also on like the function interface layer. >> Yeah, it like it gives you an extra extra layer of safety almost. you could imagine in our example earlier where the agent is like writing code to manipulate a CRDT document. The agent could get halfway through an edit and like, you know, it writes all the code and there's an error halfway through it. And so you get halfway through the edit, you apply some changes and then it fails because some contract wasn't satisfied and you couldn't type check it and you couldn't parse it at the beginning. And so now you've done a bunch of side effectful things and you're halfway through an operation and then it fails and it's not obvious like what actually happened. And so being able to have like more like parsing and and safety before you start doing a bunch of things that have side effects on all kinds of outside like software systems.

40:25 you know you're you're able to like indicate to the agent no this code isn't valid and you need to try again. >> Yeah. And like a really clear example is like this. I I'm sure everyone has some sort of classification stuff stack that they're running. And right here, you kind of just saw this where like I I didn't have feature requests before and when I go and like rerun this one, it can tell me that the categories are gone or outdated.

40:47 So if I go run this, it classifies as unknown. But you could imagine a user who's running your app and gets all the feature requests has their own category name for whatever they call. And now they could easily extend it and say new things a user wants. I can just reclassify like this should classify as feature requests instead of unknown. And it's all about this like tweaking and ownership that you really want to have as you kind of deal with stuff along the way. I'm curious.

41:16 This is probably like I'll show you the other one that I think is really weird code. I think this is the one. So like I have I have a doctor form template. Like a doctor has a bunch of forms here. It's a bullet list drop down and I'm going to extract this into the form like let's just use this example really fast. But when I actually take a look at this code per se, let me open this. How do I open this? Split down. Let's hope this works.

41:44 Which one is this one? form builder. When I take a look at this code, I and I wonder if you have code like this in your repo. where you basically have some concept where like a form, a field is described in like human terms or like your app terms. Like you obviously don't want a user to think about like string array. It's like a bullet point list. >> And you have a way to turn a bullet point list into basically what's a string array.

42:12 >> A text becomes a string. A number becomes an int. Maybe it should become a float. I don't know. Drop downs become like an array of literals along the way. It's like a union type of literal. >> Yeah. >> And then you can modify stuff along the way. But then you kind of build a new class and what ends up happening is extraction just takes in a generic. So as you're executing you actually say something like this where you generate the form type and you actually just turn this runtime type into a compile time type as you execute along the way. And now you just pass it into here.

42:50 And this basically will just guarantee that now you can just get all the fields off of it, get everything you want out of it, and now it's guaranteed to kind of give you that like JSON structure in this case. >> What you're kind of doing is because the way that like structure generation like that would normally work is you have to define a schema, right? Right. And that might be like Zod or Pyantic or effect schema or something. And like that basically gets it like it can be extracted to like a JSON schema or something which which gets sent to the model. The model generates things along that schema. And so what you're doing is basically right you're you're giving users a way to extend that schema in like a human-friendly way to then change how that extraction is done without like making the users write like a JSON schema extension.

43:40 >> Exactly. It's the same alternative as like what are you really doing here? You're just trying to go do some code like this. You're just trying to build a class. And this is like a builder alternative. If you don't do a build alternative, you get the opposite where we did the thing that I was showing the other one where like you can literally just write the code and as you you so you can kind of choose which variation you want to have but the idea is you kind of get to write the code and as you write as you write the code you can either programmatically build it which has different trade-offs like being able to build it programmatically is very useful as like code generally is but being able to just write the string itself is also useful because like an agent might just prefer to >> generate the code wholesale >> instead of doing anything else.

44:23 >> So you can kind of generate whatever code you want and then you kind of just execute it, but instead of passing around JSON schemas, we use a type system as a way to guarantee everything. So you say this thing that I generated as a type at runtime is really just turned into a compile time type for the rest of the scope. >> So So what happens if it doesn't if something doesn't match? >> You get what you would get? You get like cast errors and compile time errors. It becomes like a like here for example we have a thing that says hey if this for whatever here let's let's let's make this better is this requires oh extends so now t has to be something that extends any class >> right >> so if I do this what you'll find is something kind of interesting where this tells me this is f extends as sorry requires.

45:25 Oh, we don't have this intense yet. Whoops. >> Getting a a sneak preview of the the next version of BAML. >> Yeah, exactly. Require reflect any class and then you just do like a else throw. So now if this thing basically is not a class type because like maybe someone passed in like a array as array or something I don't what is it? Oh I have to go check what this is.

45:57 it's somewhere up here. oh array. Okay. dot array. then this would actually hit this line because this is no longer doing the same thing. It's no longer in any class type. So you get a runtime failure and you can handle runtime errors with like catches and other things that you want along the way like you would with typical exception handling. but that's kind of the idea is you you kind of control how much handling you want to do at any given point. Do you want to control this to be extremely rigid? Well, you can catch the exception. You can pass it back to the agent and say this was an exception that I got in and you can build almost like a cyclic loop.

46:37 >> Yeah. >> or you can say like nope I'll just fail the app and the user will just get no results and they'll be told to deal with it, >> right? Or of course you can choose to have your whole app crash as well. So like it's up to you kind of what level you can choose the amount of security that you want. But the idea is you shouldn't it's kind of like ad set functions like you shouldn't be forced to write things in JSON schema when an agent knows how to do something much better in the form of code >> right >> right right if the agent can write code to define some to define some workflow and it can and you can just say this is the interface I have to imply then you can kind of do what you're doing which is say hey implement this interface and as long as you implement this interface I am happy >> yes >> and I will let you I will compile it for you and I will let you go to and now if you failed to implement this interface for whatever reason so I think like an example was kind of like on the dimension of we're talking about this with the plugin gate where like I have this interface called a summarizer that I'm implementing up front and wherever I have the summarizer sorry. I'm really bad at navigating on this computer.

47:57 Wherever I have the summarizer, I actually just tell the system I have to implement the summarizer concept. And if I don't implement the summarizer concept, it will basically just fail. So I basically require that whatever class I generate has to programmatically verify that it implements summarizer. And if it doesn't bug her off, it's a compiler failure and I pass it on to the user. >> So how would how would you explain to someone the difference between like that and like using a schema to parse a generated like JSON object for example?

48:34 the main difference is like the only reason to use a generated JSON object is because what you're trying to do is you're trying to have an LLM extract or not even an LLM, you're trying to basically say that something is programmatically generated defined external to your system. But really what you I think you wished is you could take that thing as code and just pass this in as a parameter to this function and it would just work. Cuz if I had a standard type that was predefined at like compile time, obviously I should be able to pass in anything I want into T. I should be able to pass in string. I should be able to pass in like my resume class, my doctor notes class, my any class I want. It should kind of just work assuming that this prompt is good enough for that type.

49:16 >> So, okay. So, here's a question. in >> in in in Typescript for example, when you're like generating a JSON object, you might say, "Oh, the agent can generate it, but it has to match this interface extracted from a schema that we'll use to parse it." In BAML, can the agent generate a class that satisfies some constraint that actually like implements its own behavior with its own methods? Yeah, that's actually I have a demo cooking up to show that right now.

49:48 >> Okay, so I promise I promise Vibbov didn't seed that one. >> That's actually one of the coolest things that I've been thinking about a lot which is like how do we do that sort of workflow because I think I have it right here function workshop. What is this one called? Eight. Write a function get as a button implement the host interface and your class joins a pipeline. So this is exactly this. So what you have to do I think this is the demo. Yeah. Okay.

50:20 So right over here stage is actually defined external to my codebase. If you look anywhere in my codebase stage is not defined. This is just like kind of externally defined available to me. So I can't really change this. This is like read only. That's why I can't I can't edit this. If you can see this. >> Okay. >> This is not allowed to be edited. So this is hardcoded in the system. There's like special methods that I pass into it. But now I can have my own implementation for stage for trim and for read for rewrite along the way. And if I run the stages function, you'll kind of see this it kind of executed both of them. And I want to show you the code for this cuz this is probably more fascinating which is like how do we actually like go implement these cuz this code is defined elsewhere. So let's start changing this code. Let's say like I don't implement stage here.

51:18 >> You'll notice that I immediately get a compile time error. >> Right. >> Right. So, so what's happening is let me see if I can write this code and it seems take this describable interface and let's describe it externally to my code now. So render is actually a function that'll return to me a describable object. That's actually what I want to think of it as.

51:53 Render is a function that'll always return a describable object that implements this for me. Now describable is not available in this code snippet. It's externally implemented behind the scenes. But widget is going to be a thing that will implement describable. And let's just say this throw number. I don't want to deal with the throws thing. Now that widget is a thing that implements describable. How do I deal with this? Well, in my plug-in code, what I do is I actually just pass in a special thing. I say packages equals I'm going to name my package app and I'm going to pass in reflect comp reflect package.curren.

52:38 So I've now given it access to my current package >> inside of your >> inside of my snippet as the app thing. >> Right? So the you define a class in your BAML code or an interface that the code mode I guess kind of how would you describe it as like a little runtime like can access. >> Yeah. And it's not really a little it's actually the same runtime as your app.

53:09 So if I pass in my package now and I ask to get the render function. Let's run this code. All right. Oops. unresolve type my package describable. Why does this not work? I mean this works in this how my main works. So let me check unresolved type. Why doesn't this work? Oh, sorry. This is under some other name space. So it's under like app. I I pass in the whole thing. I can pass in chat 12 code mode.cribable.

53:48 So this is would you say this is kind of like in the example we went through earlier this is kind of like your quickjs bindings almost is like this is the interface that gets like injected or kind of bridged into your little runtime that's actually executing the generated code >> say that one more time. So this this package that you're you have in the plugin, this is like the equivalent of your quick.js bindings that you're injecting exactly into like in in the quick jazz case, the JavaScript runtime, right? You're defining here's the API of things you're allowed to call and what they do.

54:25 >> Exactly. Exactly. I think for some reason this is not Let me try one more thing. widget function describe for widget. Let me try one more time. Oh, I don't know. There's something weird here. But in theory, you can kind of write this code. I I'll figure out what the exact syntax is. I don't actually write Bambble by hand. I usually ask Cloud to do it, which is why I've probably up. but the idea is you can kind of describe interfaces that are described externally. So now an agent can generate a class that's dynamically defined and then kind of make this work behind the scenes without having to do any work over here. So it's like you can kind of have static comp static statically defined guarantees on dynamically generated code and then now that you have that guarantee now you can go execute this and because it's so it's actually not spinning up an isolate it's not doing any overhead so the overhead to execute this I mean you can tell how fast it's running it's basically running as if it's a normal code so there's actually no go ahead >> so what's the security model there like how do you prevent the generated code from doing evil things or restrict its capabilities is >> I'm actually curious about your take on this.

55:37 So that's actually one thing we think about. So I think the first default answer is by default you don't get it. By default you only get compile time guarantees if you do this naively and you can execute all the code and it basically lives in the same memory space as your app. So it can do anything that >> eval basically >> just like it's like it's like type safe eval >> and I think we wanted to explicitly not restrict this anymore because that's what makes us so powerful >> like making it that way. But then you're right immediately as soon as that comes up you're like okay well what if someone writes infinite y loop and blows up my server or allocates like 10 gigabytes of memory >> or postp process.n steals your API keys.

56:12 >> Exactly. Right. So there's a lot of evil things that can happen here. so that takes us to the next thing which is how we think about sandboxing and I think there's different approaches here and sandboxing has different purposes. So like there's one huge value prop to sandboxing which is like you just need state stored somewhere and for that you might need a real sandbox. You might need a true Unix environment. There's all sorts of reasons but like in your case for example you guys don't even need any sort of thing. You just need computation tasks and you just want to make sure that those computation tasks are somewhat safe. You're accessing some API. you want to.

56:45 >> So for our case though, those computation tasks, they have they have side effects, right? Because when you update the data structure, those changes are replicated out over a network and synced down to other clients. >> Yeah. And if they can hit one network request, they can hit every network request and then you're screwed. so we have a couple solutions to that. One is like one you can just build a sandbox. So you kind of create a sandbox and as you create a sandbox, you can put memory limits on it. You can put deadlines on it like timeouts and you can basically allow or deny specific like IO properties is what we call them >> and by default everything is denied in this environment I think.

57:21 >> What about reflection? Do you that I assume that has to be allowed or denied explicitly too. >> reflection probably has to be allowed and denied explicitly. I don't know what the default here is. I have to check. Probably denied. I suspect anything that's like allows you to inspect program state does this. and then what you get to do is you get this override concept. Override lets you pass in any function and replace it with an override for however you want to implement this.

57:47 And then you get the next ability which is you can run the sandbox and just run arbitrary code inside of this. >> So you kind of combine the two systems that we just described together and they kind of compose really well because you can use a sandbox without having dynamic code and you can have dynamic code without a sandbox. putting them together it gives you like the orthogonal plane that you really want. So like if you want something on the order of like let's say like let's say fetch let's say you only want to you basically want to ban all URLs except the ones that you guys allow for your like SDK layer that you want. This is kind of how you would do this. You would you would take fetch, you would wrap it and you'd replace fetch with a totally different implementation of it.

58:29 >> Okay. So let me ask you another question then. >> Yeah. A really popular use case of of this of of having agents generate code to use these SDKs is like they you want them like I said you want them to be able to hit Slack or Discord or like some other API but you need credentials but you don't want the agent to be able to yeah you don't you don't get to you don't want the agent to be able to like have those credentials in the runtime because it can like in the sandbox code reflect the runtime and and like look at what's in there. So what you what a lot of folks will do for example for background coding agents same thing if you're doing like managed background agents you have the agent running in a sandbox and there's like some network level proxy that injects credentials after the request has left the sandbox before it hits the upstream API. So, if I'm reading this right, what this would allow you to do is to basically shim your your fetch function that's that's triggering network requests and to like inspect it and say, "Hey, if it's going to this origin, you should add this credential." So, the agent doesn't have to think about credentials and it doesn't it doesn't get to touch them either.

59:29 >> Yep. Exactly that. So, you kind of get this for free over here >> without having to do any extra work, right? And that's kind of the idea is like you kind of get this sandbox layer. And of course, this doesn't prevent you from everything. Just to be very very clear, this is not the same level of sandbox guarantees that you have when you're running in a fully isolated environment that's like totally hermetic and has like firewalls baked in. Like this is running in your process. But as far as it's running in your process, what we guarantee is we guarantee function level boundaries to be completely secure. So yes, once you use shell, it gets a lot more dangerous. So we recommend banning shell. But as long as you never use shell and you don't use reflection and you ban reflection and shell, the agent can't really go outside of the sandbox. So you could replace the file system APIs with a thing that fetches against like SQLite databases.

60:19 So you have like a >> durable storage layer, >> but as far as the agent is concerned, it's just calling ls. It's just calling like all these other file system APIs in the code. >> So it never even knows the difference there >> because all of those are Go ahead. So if you did want like the same like type or the same guarantees that you would get from like an isolate or a quick.js sandbox where it's like it's running inside a separate like memory space and for example in quick.js it's it's just JavaScript. It's not node. It can only use bindings that you inject into it but it can't touch your file system. It can't like >> spawn processes or read the parent process and stuff. Is there like a sane default like here's how you get that sandbox in BAML if I need to just like execute arbitrary completely untrusted code in a multi-tenant SAS backend >> the default is just like don't allow any of these commands like don't allow any of the network commands don't allow the file access like don't allow the typical things that you want to ban them and we ban those by default it's kind of the premise and then you get everything like compute is basically what you're buying and then you also get this nice thing which is max memory and max deadline kind for free without having to think about it where you don't need an isolate to go do that. So like imagine a user runs like a while loop, we can terminate infinite running while loops with our cancellation primitives. So that becomes a thing that you don't have to go build, right? If you allocate like if the user allocates like like 50 gigs, we prevent that from happening, >> right?

61:47 >> Effectively. So you end up not having to do some of this work by default and you kind of get the wins without having to go all like part of why you didn't spin spin up a TypeScript server is you're like I don't want to deal with the infrastructure of this. >> Yeah. >> And that's and that's like a real pain point. So we said why don't you just have an agent like effectively generate code. We embed the compiler in for you so it becomes type safe by default. We and then we took another level which is we allow you to merge your types with the we allow you to graft your types into the existing types that you're compiling. So you can like effectively where' it go effectively do something like use this. So like here I can just get this demo working. I'll just ask Claude why is this snippet not working. it'll just fix it for me where like now that you have this, you can actually call the describe function which is an interface capability on this code that doesn't even exist yet when you compiled.

62:47 >> Right. >> Right. And it's type safe. >> Yeah. >> Like this thing is actually type safe when I'm executing this. >> Right. So you compiler for free you get like the things you'd have to like for example you'd have to if you wanted to limit memory or side effects or whatever you'd have to use an isolate if you want to like limit memory and CPU usage you probably have to like use croups or like complicated yeah >> yeah exact and managing process is also a lot of overhead and not only that there's latency associated with it there's like new compute requirements you need you can't use lambda functions now because of what you're trying to go do the way you scale for your for your SAS service is fundamentally different now because of this kind of work. But like what this does is this just says hey the overhead here to do this is basically a function call right and this is as slow like this compile code is as slow as our compiler is which is like go speed. So it's not like I'm not going to say it's like one millisecond. I mean maybe for this kind of code, but like for for like 100,000 lines of code, it's like maybe like 1 second.

63:56 >> Just in time compilation. >> Exactly. It's exactly that's really what this is. It's like it's pretty much like extremely fast just in time compilation. >> >> so what is what is the the the overhead of of doing this like for each execution? Like if you're serving, you know, like a bunch of different tenants for code execution, like what is like how much what like how expensive are BAML sandboxes? >> We can just know this really fast. I don't have to guess.

64:26 So the plug-in function, the render function took about 21 milliseconds end to end and the plug-in function I guess took about 21 milliseconds. So we know how fast compile took. Compile took 21 mills on my build. for this function. >> Yeah, it's pretty good, >> right? And then execution is basically just execution. There's no cost of execution. I mean, this is still a little too slow for me. I'm pretty sure why this was slow is yeah, it looks like there's some IO stuff that's I don't know why there's IO happening.

64:55 I'll have to figure that out. But 21 milliseconds is a little too slow. We should be able to bring this way down. but it's fast as far as I for like most practical use cases. it's much faster than spinning up an isolate. It's likely faster. >> So, that was my next question. It's like, what about sandboxes? Like, how does that compare to like quickjs, which is like the image? It's like a 50 like it's not 50 megs, it's like it's like tens or like low tens of of megs or an isolate like you you have a bunch of like overhead associated with creating like you know for quickjs the wom runtime and allocating the space for for quickjs in your process. like what is for BAML sandbox where you're doing like CPU usage limiting and memory usage limiting and and like how expensive does it is it and like how does it compared to >> binaries is ls that lhout this binaries like 25 megs all out >> it's tiny >> for the whole but that's including the whole compile that's including everything >> right >> right so I don't know if this tiny I wouldn't say tiny tiny would be like four megs but like or less but like h it's it's not going to be a problem and this is you never have to redeploy any other binary this is all self-contained sandboxes should also have the same exact thing they just add overhead to function calls >> basically our runtime every time you call a function of some kind once you're in a sandbox mode we certain function just swap what pointer they go to >> so you actually don't get this and like this thing that I'm showing you like the telemetry layer what we should be able to do is we should be able to even highlight the functions that who called this function, but here's the real function you actually went to >> and show you that in the telemetry layer. So, you don't even have to guess about what's kind of executing behind the scenes, right? And I think if I go back to like a previous execution that we actually did where it actually like ran trace trace trace timings do this one. Yeah. So like you can kind of get the feel for what's happening here, which is like when I call get function on this one, it's basically just executing this code along the way and then like you're able to just know how long stuff took. So like get function is basically like where did Oh, sorry.

67:18 It's basically free. Literally the Oh yeah, this is more like what I was expecting. Like the compile time is very very fast. I don't know why it was slower there. I suspect it's because I grafted types on. Once I graph types on, it gets a little bit slower >> because it has to do some weird grafting behind the scenes. But if you're not grafting, it's much much faster. So, this is kind of what I think about when I think about code mode though, like this idea of like I want to be able to write code. I want some most of the code to be dynamic. I wanted to match some interfaces that I define externally in my code at compile time. I want to be able to then use that code and that was dynamically generated but matches my interfaces as a part of my extension or my app flow and then most importantly at some point I want to be able to secure this too and like make sure that this actually runs in a very in in a somewhat reasonable way effectively because if it's untrusted code I need some way to bound this and the answer can't be like I mean sometimes it can be like spin up a dayto inbox and like go make that happen but like I don't want to manage processes I don't want to manage machines I just want it to work cuz then you have like API contracts and stuff that you have to go do and make that all work behind the scenes. Oh, I found the bug.

68:28 >> It's slow, too. >> Yeah, exactly. So, it's like there's a whole bunch of problems with this kind of workflow that I I think is like kind of problematic. And if you actually build reflection a really nice way, you get this really nice property where like you can build nice little UIs for every single function in your codebase without ever having to go do it because like the reflection defines >> what dynamically generated based on the code.

68:56 >> Exactly. So this is just like I'm just literally doing like hey for every single function in here let's iterate over it and then build a UI from like the data the types I get from here. So again, you get this like nice ability to be like like I there and this is not an LM function, right? But some of these are probably LM functions. Like this function throws something 10 / two. This returns this. But if I do this, I'll get a division by Z. And like you're kind of able to get all the reflection capabilities without having to do anything here. Like great day talking about code mode.

69:39 And like you can just summarize this text. And now this is now a function I'm executing on the fly. So you can kind of get an idea for like what you're able to do. And because everything is just code, it's just code. And that comes with all the cons and all the benefits of everything being code. >> Yeah, that's that's really cool. >> but I'm going to close it out here. It sounds like Kyle, what are your final thoughts on code mode? like should everyone do it? What are we just yapping? What's the final takeaway?

70:09 >> Yeah, I think the final take is like code mode is good for certain types of problems. If you need composability, if you need to be able to chain like complicated actions really quickly. if you need to protect your context window because you have too many tools and you need tool search. or if you're working with things that are already super in distribution like a bunch of JavaScript and TypeScript SDKs and like things like that like it it just makes sense to allow the agent to just write the code that it already knows how to do instead of like trying to figure out how to design a harness to do all that. I 100% agree.