Can an AI Out-Plan a Senior Engineer?

0:00 I think the the key thing here is when you say fighting like this is how you fight AI slop with slop, right? You're using slop to build these internal tools that make it really easy to get a really high quality document. >> And I think the inspiration that we had is very similar to async.io, but the main difference is instead of a function forcing yourself to be async.io, We want to go ahead and say that the call site determines if it's happening right if it's happening concurrently or in uh or in congruence. So the example code is like this. You'll ask the fastest model. You'll go ahead and spawn and you can name spawned context with various things. And each one of these will actually just run this code directly on here.

0:47 >> Oh, interesting. So the return type of this is a future type. Someone asked like how do you how do you keep track of everything in your head uh while you go do this? The answer is like one get good but two real answer is not get good. The real answer is build tooling so that you don't have to keep track of everything in your head. >> All right everyone. Today's episode is going to be tons of fun. We're going to go ahead and talk about how we do design docs for extremely complicated concepts.

1:16 We're going to show you some internal tools that we built of how we share markdown files with comments integrated with Slack and also talk about how what level of detail we go into with our actual design docs for a really complicated feature threading in the BAML language. Um, let's get started. Welcome everyone. Today we're going to be having a really fun episode of AI that works. This is a show where we try and show real-time effort of how to use AI models in really practical ways. This is our monthly episode where we do No Vibes Allowed. The whole point of no vibes allowed is you get to watch us code in real time. We chat about it. We share our processes and talk about something very practical that talks about both how we engineer things on our teams and how we use models uh for agentic engineering. I'm joined by Kevin. You want to give a brief thing?

2:00 >> Sure. Uh so Kevin Gregory, I've been on a couple of episodes before, but I'm a ML AI engineer at Evolution IQ where we build disability in disability insurance claims guidance systems. >> Yeah. Uh Kevin's underelling it. He's built a large portion of their agentic engineering systems while he's been doing this. And he's been really improving their uh stuff behind the scenes. He's been there for a while. And Evolution IQ, I think, was recently acquired for how much?

2:24 >> Uh we were acquired for 730 million about a year and a half ago. >> Yeah. So, you know, not a not a tiny company out there. So, >> yeah, >> pretty big acquisition. >> And then, uh my name's Fib. I work on a programming language called BAML. And today's episode, I think, is going to be one that I think isn't really done much with AI stuff, which is how do you actually build design docs using AI? How do you use agentic engineering to build various kinds of design docs? Um, I think this is something that we do a lot on the BAML team because a large part of building programming languages is actually having really good thesis and like background research on how you go do something about this. Um, and while you guys are in the chat, if you have questions, if you have thoughts that perhaps make your design process really good, um, just drop them in. But when we think about it, design docs are, and Kevin, I want your thoughts, but like I kind of find that implementation can often be oneshot if the design is phenomenally correct, >> but >> phenomenally correct design is very hard to do.

3:24 >> Yeah. No, I completely agree. I mean, we've all heard the story of the um the the guy who founded or the guy who built Cloud Code. Um from what I've heard, what he does is he basically just goes back and forth with the plan and then whenever the plan's done, he just kicks it off and then starts another one. So, >> yeah. And I found the same thing. If my design doc or my plan is really, really good, a lot of times club code cursor can get it in one shot.

3:50 >> Yeah. And I think a lot of people have spent a lot of time in the planning phases of their system. But today I think I want to talk about a what if you're doing an incredibly hard problem. I'll tell you an example of a problem that I'm working on right now that I have been working on for almost 4 days now. Um and I haven't even started coding yet. It's pure designing for 4 days. Uh the problem is threading. We're designing our threading system for BAML.

4:14 Um if any of you know how async.io works. If any of you know how threading models work and like core language runtimes, they're not what I would say the easiest thing to implement. There's a lot of design trade-offs in terms of what feels good, what feels bad. Um, and I want to show the process of how we're doing this and like how I'm actively doing this today. So, I'll show you stuff some stuff that's more polished.

4:33 I'll show you some stuff that's I'm actively working. I'll literally show you how I move forward with it. Um, and the idea of this task is I don't predict that this task is oneshot implementable, no matter how much good work we do in design. But I do believe that we could, if we design it well, we could break into like four or five different chunks that are each individually oneshot implementable and each of those could provide meaningful upgrades to the system.

4:58 Um, but before I go into that, Bart, you said, I'd love to know how you work through trade-offs and decisions where you are out of your depth. I think threading is probably one of those. I don't know how Go's threading model works. I don't know how um I have some idea how async.io works in most languages, but I don't know like definitively how it works in V8 and I don't know definitively how it works in CPython. The first process I find actively useful is actually just the ability to go ahead and have a good way to read design docs. So we actually built uh we've done a talk on this at the AIN conference. You guys will see it soon on YouTube. Um but it's this idea of fighting slop with slop. We all know we're going to generate slop. How do you do this? Well, we built tools internally to make slop really easy to understand.

5:39 So, like one of our engineers, Kai, wrote a whole thing about why we want daytime. Uh we want daytime in Bamma because daytime is nice. We want if you're building agentic systems, you want a date way a way to deal with dates. Um we wrote a whole map around it. And part of it, it wasn't actually just doing this. It was like doing a lot of background research and understanding how it's used in not our just our system, but also other languages. And you go do this. So, you build tooling that allows other users in the team to comment um like review design docs. And obviously like GitHub doesn't really work well for this cuz GitHub's not built for like sharing a massive amount of markdown files really easily. So, we added this little bit of tooling. Then we went ahead and added a little bit more tooling to actually like connect all this to Slack. So every single time stuff gets created, a Slack thread gets created for every single thing that comment that uh reflects it on here because again we don't have notification systems on our website. We don't want to build that. So we latch on to Slack as a notification system to make sure that design docs can actively be shared once they're in like a more ready state. Um so one way to kind of deal with decisions out of your depth is how do you involve more people in your team into it? And you have a couple different options here. And the easiest option in my mind is just make sure that people in the team can read it. And some of these threads, let's see if I can find a good one.

6:55 Um, and like we don't always use this. Sometimes we just use Slack directly. But often times people just read these and like we'll just start leaving comments and we did extra work to like tag the person connect the person in Slack to the person in our system over here as you go do this. And like we actually built the and again most of this stuff is hard like we can actually see all the users here and like some people are like have different privileges like me and Aaron have slightly different privileges. Um since um and everyone on our team gets automatically connected because if their GitHub account has their boundary ML email they automatically become a member of the team and members of the team have different privileges than like random wild users that want to go do this. So like for example if you guys go to bs.boundaryml.com boundaryl.com. You should be able to log in when you're GitHub and just see random work that we're doing. Uh but yes, this tool is completely in house. We haven't really um if you're curious, it's actually fully open source as well. If you go to our repo um uh where is this TypeScript 2? Somewhere in here. If you just ask Claude to find it, it's somewhere in here. I don't know where. Yeah, this is this is really I really like this idea because some of the one of the big things that I think that that we struggle with and I imagine a lot of other companies too or a lot of other companies do as well is you end up in like design doc help right? So, we use we use Google Google Drive a lot.

8:17 So, we have Google Docs kind of all over the place, and we don't have a good way of of tracking what design docs are being discussed, what's been approved, what comments are are where. You most of the time people kind of just send it out. There might be one round of comments and people reply, but there's no like sense of when you have a PR, it's merged and it's done. There's no sense of that. And so, something like this, I think, could be really, really helpful. Really helpful. I actually might might steal this.

8:45 >> The one I'm working on right now is spawn cuz I'm building concurrency and like they have different states on here. You can mark special things as like good for the LM. >> So then other L other things working at new design docs automatically pull them into context as like reference. And I'll show you how we do that in a second. And then the other thing we have is like this export ability where you can just like export things. Um but yeah, it is effectively a tool to uh to just um kind of like be able to leave comments and share information about them. Now there's a big assumption in this tool which is the person that's producing the design doc once they move it from draft over to proposed actually goes ahead and has done the leg work to say that it is good and it is it is good to read. I trust you uh to go do this. Um we haven't built a notification system where you can like ask specific people to review. That's kind of a pain in the ass.

9:38 >> Uh but like we just tag people on Slack and say hey go read this. But um now we're taking this a few steps further because again the problem with any website back system is if you're doing a website back system, Claude can't edit it. So we have a thing that allows you to export all these beeps that gives you a nice little zip file. And when you have the zip file, what you get is you get a nice little folder structure that downloads every single beep and every single version and uh gives you all the data about it and all the pages on here.

10:07 Now, the other thing I've been building on top of this next step is actually some cloud skills. I haven't checked this in yet, but this is a cloud skill that has like another CLI tool that I've been working on. That's a Python script. And the whole idea of this tool is this. Watch this. Clear. Uh oh, I just spent some tokens. Oh no, that cost me money. It's really interesting just typing things into your seal internal now. just randomly bills you. Um, but it's not really an empty U. So, let's let's do the next thing. So, what I'll do is like pull pull the data. So, when it pulls the data, there's this CLI actually runs. There's like a cloud skill called BEP. Uh, let's see if it auto uses it. Nope, it didn't use it.

10:59 Beep pull. I can say beep pull. And what this will do is it'll actually just run the script and this will pull the data and it'll tell you something is out of sync. Uh so let's change this really fast. I'm going to go ahead and like change the script to like add some new data. And now let's run bp pull. And now you'll notice it's going to pull the data and actually tells me that this thing has two lines removed from readme md. Like there I guess the diff is wrong. So I should update the script.

11:28 Oh, if I pull I'll remove two lines from readme.md. I can even ask which two lines. And because this is all backed by Claude and Claude is using this, I'll show you in a second what the B app pull actually shows you. >> So this is making sure that your local folders, your local bets are in sync with the what you were showing us earlier in the UI. >> Exactly. Because we don't want the problem with using Git for this is then you can't build all the tooling that you want around this because Git doesn't have a good way to really guarantee certain kinds of tooling. So it actually, as you can see, I'm I'm just working with claude to ask it which two lines. It just did the thing. It pulled the thing. Now I say, uh, yep, just just use the cloud thing.

12:13 >> Cloud thing. >> And this will just kind of do the do the thing for me without me having to do any more work. And like boom, my apps are now up to date. >> U, and it does all sorts of things like renaming. It's kind of robust for this. And this is kind of where I think the blend of software versus hardware of software versus AI really comes in. I worked with Claude to write this script.

12:31 I haven't I don't even know what this code is. I don't care because this this code is a means to an end. And this is what we mean by fighting slop at slop. You generate slop code. Don't really care what it does. As long as this workflow is good and this is nice, I'm very very happy with my life. And this workflow is I can just say like I want a concurrency beep. Let's go work on this.

12:54 And then what I can do as a developer is I can spend all my time working with Claude on a concurrency system. And we'll talk about the concurrency system in a second. Claude can be editing this for me. I have to spend zero time thinking about this. I can do all the background effort. I can do all the effort around understanding how current currency models work. Uh, and then I can write a bet for my colleagues to go review and read and they can read on a nice little UI on a dashboard while I can edit with a markdown file with Claude. Does that workflow overall kind of make sense, Kevin?

13:29 >> Yeah. Yeah, it does. And I think that the key thing here is when you say fighting like this is how you fight AI slop with slop, right? You're using slop to build these internal tools that make it really easy to get a really high quality document. >> Exactly. Yeah. >> And that's okay because it's not it's not customerf facing. It's a pretty simple workflow and it doesn't matter if it's sloppy or or doesn't follow solid principles or whatever. If it just gets the job done and it helps you get to this state faster and easier. So then what you actually end up shipping is a lot better and more reliable then that's that's a worthwhile trade-off every time.

14:05 >> Exactly. The for those curious if you look into the BAML repo you'll find the beeps folder. That's kind of where this is. Um, yeah, I would I don't think I've ever looked at the code in the beeps folder. It is a pure AI slot mess and like I have we the only way I add features to beeps is via slack and tagging coding agents to go add features. I have never even opened claude myself to add features into beeps because it's not worth it. If folks have more questions about this workflow, let me know. Uh but otherwise I want to share like how we go really deep into a really hard problem. Uh any question on your end Kevin?

14:45 >> No nothing for me. Seems like the the chat people are ready to get into the threading. >> Okay let's talk about threading. Uh this is a super super preview. So if you guys have opinions share them as you do. So let's start off with the problems of threading really fast. And this is how we start. The first thing that we do when we often write BAPS and at this point we've gone through like many versions of BAPS. This over this actually overrides like this previous version that some Antonio on our team wrote. Oops. Um we're like at least for me the worst why do we want threading?

15:21 Well, if you're writing aic workflows, you're writing any sorts of systems um oh what does BEP stand for? BEP stands for BAML enhancement proposals. It's a way to add new language features into the BAML language. Um, so when we think about threading, I think the worst worst worst part about threading is actually the fact that you have colored functions everywhere. Most people don't do threading. We've decided as a society that asyncio is more convenient than threading and easier to model for most people than threading. So we do async.io. And async.io That was a really nice system that allows us to get par I wouldn't say parallelism but rather concurrency because it doesn't run things at the same time. It actually runs things just once. And let me see if I can find a doc over here. It's on here. Uh the problem with async.io However, is that if you've ever used TypeScript, you will often see a function like read file sync read file async because once you are in a async context, it is really hard for you to leave and go into a sync or once you're in a sync context function create user.

16:31 If you if create user was a database call, you can no longer write a weight here unless you mark this function as async. And I think that painoint exists almost as a legacy painpoint. Um what is it? TypeScript. And I think the reason that this painoint exists as a legacy painpoint is because concurrency was not something that most languages had on day one. So if you don't have that, you now have two code bases. And many times I have run into this problem where somewhere deep in some nested stack I need to use some async function. And now I have to freaking wait and I have to change the whole stack upstream to make it completely sync. Have you done that?

17:10 >> Everything's got to be Yeah. >> I I've I've done it a couple of times. Yeah. Everything has to be Yeah. You have to change it all the way up and down. >> Exactly. And I think when you do agentic engineering and we want code to happen, >> you want to have the minimum amount of diff to make sure that the right thing happens in the right way. So that's one of the problems that we are dealing with is we don't want function coloring. We don't want to have an async version of the function and a sync version of the function just to support how our callers might want to use it. But we do want parallelism because if you're calling an LM, if you're calling five LM functions, you kind of want them to run in parallel when when they can work in parallel.

17:50 So that kind of inspired us to think a little bit more. And I think the inspiration that we had is very similar to async.io, But the main difference is instead of a function forcing yourself to be async.io, we want to go ahead and say that the call site determines if it's happening, right? If it's happening concurrently or in uh or in congruence. So the example code is like this. You'll ask the fastest model. You'll go ahead and spawn and you can name spawned context with various things. And each one of these will actually just run this code directly on here.

18:29 >> Oh, interesting. >> So the return type of this is a future type R2-D2. Why is superseded in the new miss missable row? Uh it's just a slop artifact and we don't really care about reading um reading superseded. So it's not a thing that has really bothered us. Um, and you can see over here each of these is a feature and then you can await any of the features and then you get the first response back. The other thing that we want to be able to make really easy and I I should help that what helps us design these systems is actually starting off with one of the premises of BAML is to be a great language for application development. So when we do design work on here, we always think really hard about there's a cloud MD that has some rules, but effectively effectively the rules that we have are like think really hard about what is a frequent behavior and the things that are the most frequent should be the syntactically the most convenient while not compromising correctness to some degree. And if uh correctness does have to win to some degree, but frequency is really important. We don't want to make it harder to do the right thing.

19:38 That's important to us. So, one of the things that we realize that a lot of people want to do is thread pools. If you want to run an array in parallel, you want to kind of say that, hey, this thing is running on like I oh, it's a it's not supposed to be this. I have a different version of this, but his idea is you should be able to say that I want to spawn things and run at most three things at the same time. So, we have this concept of a Q.

20:01 But the basic way that we did this work and you'll see kind of how I do the inspiration for this is especially now that this beep is getting more and more complete is we kind of have to come up with some analogy to some existing system. So, we've done previous leg work to recognize that where what languages do async await, what languages do virtual threads, what languages do OS threads. We know we don't want to do OS threads because they're extremely heavyweight and really complicated to get right. Um, and most application developers don't want to think about OS thread levels. You don't want to be thrashing your threads.

20:34 We do likely want to copy Go or Cotlin who have co- routines and many languages Python has co- routines etc. and go make that work. And again we don't want async aait because async aait leads to the problem of coloring where we have to label every function as either async or non async. And if you want to use fetch now everything upstream must be async. So we want to avoid that problem when possible. So once we started with that we basically just forced the model to go ahead and and I'll go through this in a second. Every single part of this beep has to kind of be written in a way that is somewhat readable. We invented something called middleware that allows you to do things like wrap a spawn with a retry over here.

21:15 >> And that's kind of convenient because >> many times you want to be able to just retry arbitrary blocks of code. >> Uh you want to be you might want to say that a spawn has like I'll talk about a few more examples >> a fallback where if it fails just give me a value and that guarantees that this fun this future can never error anymore. the error type is never as opposed to whatever it was given to be and a few other options that we came up with.

21:42 We'll go into this in a second, but as we go through this, one of the things that you'll notice about this beep is that it's extremely thorough and complete with the examples. Normally, I'd be really lazy, but I don't have to be. I can literally say like, "Give me an example of retry. Give me an example of timeout. Give me an example of timing." And what does timing do? It takes the spawn and every time you run it, it just logs how long it took to run the task name, the name of the task or uh that's given to it. So in this case, it would just log how long the extract took and it tells you like it'll run the retry and with the retries it will log the timing of the total system, not each individual retry.

22:19 >> Oh, cool. >> And obviously like with retry and with timing is different than with timing with retry. This measures the full system. This measures timing of every single retry individually. And one of the things that you'll notice when I go through this is there's examples like fire forget. There'sam and I I'll read through the beep a little bit more slowly in a bit, but I want to show the process first. And I want to show the level of thoroughess that we go into in here. We talk about unhandled spawns. We talk about how how futures that spawn futures work, especially for example, if we do cancellation.

22:54 Uh we talk about rate limiting. We talk about the cancel token and then we go ahead and like talk about how you do conditional spawning how select works for example like if you want to pick one thread or the other see which one got completed but the point is this docet's very very thorough now once someone reads this it's we found a what we do is we actually record the slack meeting using transcriptions and I'll show you the meeting that we had about this talk recently it's like a giant transcription um language.

23:28 Where'd it go? So, we literally just recorded. We had a Slack huddle. We got the notes from the Slack thread and then we actually just have the huddle transcript. I The notes don't really matter, but I literally would take the full huddle transcript and we were in person, so it's just that's why it says just me talking. >> It's just you talking. >> Uh and it's a pretty long meeting as you can tell. Like we were talking for like an hour and a half here. Yeah. Um at least maybe two.

23:50 >> Uh two and a half. I don't think we recorded the whole thing sadly cuz granola broke on us. So I literally just went through I copied this whole transcript and after I copied this whole transcript I literally say something like this beep 34 is very complex. We make a and I literally just reorganized this because I realized that this beep which is spawn is implicitly done very it has so many design decisions that we have to make uh like cancellation like cancelling threads and cancelling workloads is a whole complicated workstream. we have so many design decisions that we have to make that even someone reading the BEP doesn't have the full context and I think I paste it in the trans at some point I do paste in the conversation and I basically just force the model to go ahead and just sprint out an outline of how it should rewrite the beep and this be I want to say the summary the motivation the simplest form the design decisions and this time it outlines all actual syntax decisions that we make like our future sharable can you like send futures across threads themselves. What happens when you await on a future multiple times? What happens when you What happens when you throw?

24:56 How are cancellations taken? The fact that a parent being canceled means that all children get canceled by default and you need to do work to detach themselves. Um can you have a thought then on a future where you actually choose what it does what it does in different situations? So these these are these are decisions that you discussed in your meeting or that is implicitly decided in the document >> both. So some decisions got changed and got introduced because of the meeting and some are just locked into the document.

25:30 >> Gotcha. And then what we did is I basically asked the model to look at these design decisions, look at and then uh pull out the more complicated ones and then pull out a whole bunch of examples over here for each of these and then just call out what we're explicitly not doing because that's important for people to read at the uh at the map of like here's just like I'm not talking about test local storage like thread local storage is not in scope of this thing. We're not we actually remove select after talking about this design decision. Uh, conditional spawning is just like we don't it's just a little complicated. It's not relevant of putting in here. And deadlock detection is something that we can do, but it's not something that we're going to talk about in this map. It's just out of scope.

26:10 >> So like having a really good philosophy of what we do ends up being very useful. And what we end up doing is I'll show you the final optimization for this was actually like I want to reduce the scope of this beep to be much smaller and much more direct. And the final thing went from like 104 kilobytes is how big this total beep was down to 62 kilobytes. So I reduced the amount of like verbosity by half and I kind of have to go read the whole thing to make it actually good. And I'll show you what the final thing looks like over here.

26:40 And now you can see that the beep 34 V2 and I made it right in V2 because if I delete V1 what you'll notice is it will if you replace in place for design docs models will often just mess up >> really. >> Yeah because like >> that's really interesting to know >> for complicated design docs I've seen this 100% of the time and if you think about intuitively it makes sense too like why would a model think about a human humans get lazy and they're inconsistent when they edit things. Edit editing is a more hard exercise to be coherent in than rewriting from scratch.

27:14 >> Yeah, that's a good point. >> That's a good point. >> Like imag take any software architecture like take any agentic system you built. Think about how much cleaner you would write it the second time around than the first time around. >> Yeah, I think it's it's something similar where when you see um a system that has a lot of tech debt, the there's that part of you that just wants to rewrite the whole thing from scratch rather than kind of just editing it.

27:34 Right. It's just it's the same thing. >> Exactly. Um, and I think there's like the sunk cost fallacy that a lot of people have, which is like, oh, I'll just edit it. I'll keep editing. But often times when you're doing like in this case, I'm effectively doing a major rewrite where I want to like re I want to be like, hey, spawning is way different than every other BEP that we've done before. It has so many more implicit design decisions that are being made that are not obvious. I want to just label them one by one by one and then in a separate document talk about prior art and like how other people do it.

28:06 So the first document was the first document combining both of those two. >> It was literally just interweaving all the design decision all over the doc. And >> so that's that's how you were able to get it from the larger to the smaller even though you're saying >> here are like discuss all these decisions in more depth. >> Exactly. >> Because you're splitting it out into two different ones. >> Exactly. And like now if you read the spawn doc I'll show you what it starts off with. It still has a motivation section cuz every time you propose a language feature, there should be a a user value here.

28:37 >> It very much highlights function coloring as like a very top level priority that we have, which is we don't want function coloring. >> Um, and then it just starts off with the very simplest forms and the na um you're not keeping all the versions. So, I'll talk about versions and how we deal with versions in a second. We talk about the simplest spawns and all the name spawns. >> Mhm. And then we literally just start off with every single design decision and we talk about why. Uh like when do spawns start? Do spawns start when you hit a wait or do spawn start immediately as soon as you spawn? That's a choice >> or do spawn start explicitly when you hit dot start right like threads don't start often until you hit start >> uh in a lot of libraries. But in our case we've decided that spawns actually start completely immediately as soon as you hit spawn because why wait? Um a future is sharable. So once you have futures, you can actually like await something twice. It's item potent. It gives you the same exact response.

29:33 Futures actually outlive their spawners. So you can have a future that gets returned by a function. Why? Well, that's just useful for marinating paradigms like map functions will do this. If you want to take an array of URLs and like run them all in parallel, well, you make a future. >> Um, we had a choice. Do we want a weight to be in front of the thing or do we want a weight to be a postfix like f.8 Oh wait, like Rust style and like our target audience is Python and Typescript devs. So we prefer looking like TypeScript. But if a lot of people end up doing a do have like chained awaits which often like if you're writing like you'll run into this if you write a web system which is like await fetch dot uh JSON await. you have to like double catch your awaits over here if you do this because the first one gets like the metadata and the second one actually gets a payload.

30:27 >> But like that's like one edge case. So we're okay with that pain since it's already familiar to Python and TypeScript apps. Um await rethrows errors from futures. So if a future has an error, a wait just throws the error of the future and it's completely type safe. Um cancellation is a panic. So one of the things in the BAML language that we have is errors are completely type- safe and we infer whatever error message error type a function can throw regardless of you doing that. The problem with inferring error messages and having like exhaustedness on errors is it's very easy to have a wild card accidentally uh like hide a cancellation.

31:07 >> So we have two kinds of error messages. One is like an error that you deal with. one is an error that you um that you kind of have to like explicitly catch. If you want to avoid cancellations, you have to explicitly say no, if I get a cancel signal, ignore it and give me this value instead. But by default, it'll just get rethrown. Um cancellations when cancellations happen, when a points happen. But I think the big difference you can see, let's see if I can pull this doc side by side. One big thing that you should be able to hopefully see almost immediately is like just like how this doc starts versus this one.

31:44 >> Um, and like almost immediately there's way less pros. I think I'm zoomed in more in one of them, but I'll zoom out so it's the same size. Almost immediately there's way less crows. Um, >> there's the uh mental model is kind of like garbage, so I got rid of that. The motivation is way thinner and way easier to read. uh in my opinion, it's just less text like size-wise. >> And then it starts off directly with like just like the very very basic example talks about the most common use cases, the fact that you can name stuff for debugging use cases. Um um and then it goes straight towards like like the previous example just started talking about middleware. Well, why are we going to middleware right away? we should talk about the design decisions that we actually made and it's way easier for someone that's just skimming to digest it.

32:35 >> Yeah, I think it's it's important to remember that the models tend to read this a lot like a human would, right? And so if you just jump into the kind of an immediate rest, you start with something very specific and you don't have this like layered top- down approach, it's going to be a lot harder for the models to understand >> exactly implement. So we spend a we spend a lot of time this thinking about how we're going to go have a model think through this. And once it helped, >> this is probably one of the most complex design docs we've done to date, >> which is why it's very different. But we did have a set cancellation if anyone's ever tried is a really really hard concept to go model. But for us, we know our target audience. It's people like Evolution IQ who are building massive agentic workflows. Well, we know the default here, which is if you're actually going to go ahead and cancel like I'm an app developer. I spawn an API that I I spawn some library code that does deep research and spawns like 500,000 agents to go do stuff and something comes back to me and gives me a result faster. I kind of want to cancel all the work that that that research started and just kill it and who cares what that thing said. And so cuz before API calls didn't really cost money.

33:48 >> Yeah. >> And now like every API call >> it's a tool use. Yeah, >> it's a tool. Uh, exactly. It's money. So, you kind of want the right to be in the app developer hand to decide when they cancel work. And I think at the bottom we talk about like prior art and what happens here. >> Mhm. >> Um, oh, it's not enough detail. Okay. So, this is like one quick readout here. I immediately see that this prior art section is very weak. So, what I would do is I'd say I want a subpage on prior art about design decisions that we made on a board controller, for example. Like a board controller is probably the best example. So, let's I'll show you exactly what I'm going to do. This is pretty good, but the biggest miss here is a lack of understanding for the end user on why we didn't go with explicit cancellation tokens. For example, like go or controller in Typescript.

34:42 Obviously, there's a syntactical error and both languages have made different trade-offs. In the case of Go, every function has this thing called ctx. So, if you're layering things through like 17 different layers of functions, every single one of them well now has to carry ctx and pass it down. While this is technically more explicit, it is a burden for app developers that are first being welcomed into the language to just have to know this magic parameter and they later learn that it's about cancellation. and we want to avoid that burden. On the second hand, TypeScript has a different philosophy. There is no philosophy around passing in a cancel token. So 99.99% of the time, no one uses an abort controller and no APIs in Typescript are ever cancelable by default. And no library has cancellation semantics really built in. And we don't really want to be in either of those worlds. So we prefer the the implicit cancellation of Python, for example.

35:38 You'll notice that I'm not actually trying really hard to teach the model anything here. I'm very explicit in these in this learning. Uh make this a subpage. I'm very explicit in the learning here. Uh because what I don't want to do by accident is I don't want the model to really make its own inference. I will ask it about its own inference once it's done. But I wanted to really capture the thing from the design discussion that we had more true to myself, >> but I'm not going to put it in the main read me. I'm going to make a separate subpage about this because I know for someone that's new to uh reading the this beep. They will probably prefer like why we didn't pick existing semantics in a whole different page because it is somewhat nuanced and detailed. And we likely want code samples about this. We've got a couple questions come through in the chat. So, one is about versions of all these different documents. Do you keep the different versions? Models go nuts when they see multiple versions of something.

36:38 >> Yes. So, we actually have two different ways of working with beeps. Um, one is this what I showed you where you download all the beeps and you work off of them. Um, because you kind of often need context of other beeps to design other beeps. >> Mhm. Um the beeps are not usually designed in independence but the other approach we have is actually this approach. Let's say I'm uh we're working on reflection for example actually this is approved uh we're working on reflection. You'll notice that we do have versions built in. Actually let me pick one that actually had a lot of versions patterns. We're working on pattern syntax. There's seven versions on this beep. Every single version of this beep has its own comment chain. Has other things driven by it. There's a quick little thing to remind you, you're on an older version. Uh, you can't edit comments on old versions. They're read only. You can't see them ever again.

37:29 Uh, but if you export this beep, I'll show you what we do. We actually, when you export just a single B, you actually get all the versions baked in place. >> Uh, you also get all the discussions and all the questions that people have and you get all the comments and everything baked into agent context.md. >> That is very cool. So, >> so sometimes if you're working on a map and you want to refer to other versions, then you have to go through this workflow. Ideally, we can merge the workflows, but this is the problem with slot based design like you kind of have you kind of just do what you need to do at any given time to make it work.

38:01 >> Um, but this is kind of the approach uh for like versioning. You do want versions. It's useful for humans. It's useful for agents. But the reason that we don't use git is because you often like one comment tracking is really hard on git uh for like various diffs once you start doing diffs and also we want a very linear history for our beeps. It needs to be like purely linear. You push to it and that's it. Um so the versioning story is slightly simpler and that's what works for us at least. We might switch to a git based approach eventually um but at least for now this works well.

38:37 Yeah, I I'm curious how much because we're spending a a lot of time and this is kind of what we talked about up front with how important it is really really getting a good design doc now because you can almost oneshot it. Maybe not with threading but how how much more time would you say you've spent now doing this kind of work than you know 2 3 years ago? >> I think I'll show you an example of a beep that I would not have written in nearly as much detail without without this. Um, one of the things our our middleware BEP for example, >> I want to show how many examples we have in the middleware. Our middleware BEP, which is like a way to add middleware into the system. Like you want to say that this scope of code has a cost limit of $5. That'd be nice to have >> like don't spend more or like hey use a cloud use like the cloud SDK with the string passed in >> or run like a retry with a timeout on this fetch. Um, it's kind of like our middleware >> BE >> and I want to show like how complicated and we talk about all sorts of things like why don't you do wrapper functions and everything here too.

39:42 >> Um, but when we write this one of the expectations we have is like this prior art I want to see code snippets of like real systems and I just I would have been lazy. I would have said express has this. I would have said Python decorators have this and Python decorators I know off top of my head as well write the code. Uh I but I no way would have found like the poly.net mechanism of writing middleware. I don't know. Net so it's just not something I think about. Um so I think there's small things like this that would have made a big difference. And then and when you actually go down like I wrote all sorts of middleware here to prove that it works. I wrote like a with retry retry and I actually wrote out all the code. Then I went ahead and implement the timeout and timeout uses spawn and because I have all the beeps in context it can actually go do that and write how timeout would be written. Then I wrote timing. Then I wrote fallback. Then I did composition.

40:38 But then I started doing more advanced things. What if I want a retry that has a backoff of a certain type or you have exponential backoff or like jitter or constants? Um I'll if you want to read the beeps you should go to beeps.boundaryml.com. boundaryml.com. If you want to find uh if you want to re see the beep repo that's in the baml repo we have a monor repo pattern but then I did selective error handling.

41:03 What if I want to retry on only on certain errors? Well like now you can pass this in and your code looks like this. You do a you you're running this code called fetch with this API named this section of code named fetch with API calls. It has a retry of three and it'll only retry timeout error rate limit error. everything else will not retry on and just throw the exception. So like authentication errors will not run the retry loop.

41:28 Um and then you we built a circuit breaker which is like it's kind of like a rate limiter but slightly different. You can look into the pattern later if you're curious. Then we built a rate limiter. Um then we went further and said how do you compose different compositions here? And just this level of example building is just not something I would have ever done before. There's like zero time I would have spent on like >> doing this. I was like, I built a caching system. I want to say like, hey, run this block of code with a cache with this key.

41:58 And again, I would kind of know it works, but the point of discovery for whether or not there's a bug here would be much later rather than earlier. I'd discover it during implementation, be like, "Holy cow, we have to redesign this thing." And I like the basically the best engineers would make less skill issue problem would have less skill issue problems. So their implementations would be better because their intuition is better. >> Mhm. >> But now like everyone everyone's median kind of rises in my opinion and your median is so much better than it used to be.

42:32 >> Right. For sure. >> That's fascinating. >> How much time do you spend on writing beeps? I spend like or not beeps but like writing design docs. I spend a lot of time like writing design docs and plans for almost all of my work now. It's like >> 50% >> I think. Yeah, I would say I would say it's more than 50%. Most of my time I spend writing docs, coming up with plans. I like to keep it I heer on the side of more detail. Um and I think it's similar to kind of you know what we've seen. I'm not going into more detail than you're threading one, but I I spend most of my time reading design documents and plans and iterating on them and um because the code you kind of just again if it's good enough, you can kind of just oneshot it. So you just send the design doc and the code kind of writes itself and then you review the code and or and then and then you merge and then you're done.

43:26 >> Yeah. So now that the the job of hands on keyboard typing code is kind of just been >> solved, >> we I have a lot more time to do to write these design docs and it's so much more important to do that since you're not writing the code. You have to, you know, if you're giving instructions to someone how to do it, all the stuff that's kind of in your head that you or assumptions that you've made, you have to make sure is really explicit in the doc. And it also helps question >> And it also helps um question your assumptions, right? Like it comes up with >> it finds that you're assuming different design patterns and things like that that you didn't realize you you didn't even realize that you were assuming and that that might not be best.

44:10 >> Yeah, exactly. Like the cost limit one is kind of interesting. When I was doing the middleware, I was like, "Oh, I want to build a cost limit here where I just say like this thing runs and I want to spend at most $5 here." Well, in order to implement this, you have to implement a thread local variable. Like you just need thread local storage. There's no way around that. >> Mhm. >> Well, if you're going to do that, well then like there's really not much around this except doing that. And in order to go make that happen, well then it's kind of your responsibility to discover this problem. And it might have been impossible for me to have thought about that really hard and said like oh oh holy holy cow we actually have thread local storage >> and like >> but LMS like LMS will write every piece of code that you ask them to. You will you can say I ch I want you to challenge me with what should not be possible in this design but is going to actually be done here. I think this is also something that you brought up in a previous episode where you and Dex were talking about. Um, it was if if you don't if you just tell the LLM something, it's going to assume you're correct because they've been trained to to to basically to trust you and that you have context that they don't. And so something that's really helpful is is almost like a here's what I'm thinking for something, but I'm not sure what other ideas do you have for this design pattern or this part of the system. I found that to be very very helpful because it will just assume what you're saying is correct and then it'll implement it when it may not be.

45:36 >> Yeah, exactly. I think someone asked like how do you how do you keep track of everything in your head uh while you go do this? The answer is like one get good. But two real answer is not get good. The real answer is build tooling so that you don't have to keep track of everything in your head. >> Like >> the fact that we built this tooling let you download every beep and go do this.

45:55 I don't keep track of everything. I write the BEP and I literally say, can you go check every other implemented BEP and see if we are if we're consistent with it and if the syntax is correct and if there's any like weird interactions. I do try and like have my own model of it, but these things are nuanced and they make a lot of mistakes very easily. Sam on our team just make it a really good suggestion. We used to name our BEP folders with just the numbers and now we don't. Now we name them with numbers plus the name because if you do ls and the model does ls, it sees >> exactly what dep is without having to read anything.

46:31 >> Um, and just constantly reinforces where it has to go do the work. So I think there's small kinds of tooling that you can build along the way to make this really really helpful. But I mean that's it for today's content. If you guys have more questions, happy to stay on and help answer them after the fact. But I think that's it. Kevin, thank you for joining. >> Yeah, thank you so much. Hopefully you guys got some interesting insight on the tooling. If you're interested in checking out, go to beps.boundml.com. Or if you want to go read how the code works or don't read how the code works, ask Claude to read how the code works.

47:01 Uh check out a GitHub repo and ask it to the uh check out uh GitHub and ask it to like ask Cloud to say where where's the beeps folder and how do I run it? It'll get you set up and it should do everything for you.