The Data Structure Problem AI Coding Agents Can't See

0:00 If you've chosen the right data structures and organized things well, the algorithms will almost always be self-evident. >> Most people end up building their own janky sync engine. We'll talk about what it looks like and and all the things you will end up having to think about when you want to optimize for performance. All slop starts as bad data structures. And if you can simplify those, you can simplify downstream code and be more confident in it.

0:21 >> This was a really, really fun conversation. I hope people that are watching today got a gist of a slightly different kind of episode today where we actually got to go deeper into like not just me and Dexter happening but some technical conversations and you got a little bit peak behind the scenes of like how people on our team engineer and think about data structures when building compilers and how people on the human team think about building distributed systems that can have really fast syncs for complex UIs that need to be fil native.

0:49 I think everyone watching this episode that hasn't watched it yet is hopefully going to go learn a lot. And I and like if you haven't already, I'm gonna say before you get into the episode, definitely go ahead and go follow Kyle and Avery on Twitter. If you don't know their Twitters, I'll share it really fast. Go take a look. Kyle makes some banger tweets all the time. Like how Jared Palmer is apparently still still at Versel even though he technically works for Cognition. and like Avery makes some u Avery's got some funny tweets all going on all the time. I'll show you my favorite one really fast. What is your prime one >> right here?

1:27 >> They really got this going and we they I don't know how they missed this, >> but go follow them. You'll get some really good engineering content from all of them. and like hopefully today's discussion was if it was a good style that you actually enjoyed. let us know in the comments and like we'll try and see if we can replicate this kind of conversation going forward. But other than that, let's get started. Let's go learn about data structures and let's go learn about sync engines. This is the AI that works show where we talk about AI that works in production, beyond the demo, systems engineering, AI engineering, learned in the trenches, working with hundreds of founders and engineers who are shipping real AI products that make real real revenue.

2:10 I'm here joined by my host Vibe Vibe Code Vib and his guest. You guys want to intro yourselves? >> how's it going folks? I'm Vivof. I'm one of the co-founders at Boundary and we build a programming language that's built for the AI systems. >> I'm Avery and I also built that same programming language working here at Boundary for the last roughly six months. >> Some of you may know Avery from reclaimed fame of pursuing his battle against Microsoft for $5 and finding exactly what what Microsoft's willing to do and what source code they're willing to send you.

2:43 >> The the letter is in the mail. We're still waiting on that. >> Okay. Today's episode, I think, is going to be a really fun one. Dax and I have been chatting for a bit and obviously some of you saw the unconference. We saw a lot of people like the comments on it. So, today's episode is actually going to be a little bit more of a recap from the unconference. We're going to have Avery give us talk which is about data structures. I won't spoil any of it.

3:04 we shared a little bit yesterday, but last time, but this week we're going to get into it in really big detail. And I think we're going to have Kyle hop on and talk about how we do syncing across like multiplayer systems. when we're building agentic systems kind of like Dropbox but for more for AI coding workflows. Is that a good summary this >> I don't know exactly what he has prepared but yeah we're going to talk about the sync engine we use and how you might accidentally reinvent a sync engine from first principles and why there you should you know like many things you wouldn't build your own database unless you were crazy but most people end up building their own janky sync engine and so we'll talk about what it looks like and and all the things you will end up having to think about when you want to optimize for performance and what what state of the art offtheshelf tools can do to help.

3:55 >> So, this episode probably Dexter and I will just shut up and watch amazing people actually talk. So, if you guys have questions in the chat, just drop it below. We might ask some questions, interrupt you sadly because we're obnoxious. >> and Kyle will get the same treatment right afterwards. >> But, we're going to hand the stage out to Avery and let him share about data structures really fast. >> Okay. So, I'm going to start off with a selection of quotes. And hopefully this this convinces you that at least in the past, people cared a lot about data structures. Fred Brooks, show me your flowcharts and conceal your tables, and I shall continue to be mystified. Show me your tables and I won't usually need to see your flowcharts. They'll be obvious. He's basically saying in in modern parliament that if you if you just look at the control flow it's it's quite confusing but if you if you look at the data structures behind the control flow it it will all make sense like intuitively >> and really quickly who's Fred Books >> very famous computer science guy.

4:54 >> Okay, cool. >> What was his quote? It was like adding more people to a team slows down the project. It's from the same book that comes from. Yeah. >> Yes. What was the It was like Yeah. Yeah. Nine nine months to a baby, but nine women, you cannot make a baby in a month. >> Yeah. Yeah. Yeah. >> Yeah. >> can't say that anyway. >> Yeah. >> >> I mean, you know, the future is here.

5:21 >> Rob Pike. >> Rob Pike, the Golang guy, as you can see on his shirt. If you've chosen the right data structures and organized things well, the algorithms will almost always be self-evident. Eric Raymond's full knowledge into data so program logic can be super robust. And then if you haven't heard of any of those other people, Lionus, bad programmers worry about the code. Good programmers worry about the data structures and the relationships. So what does this all mean? I'm going to I'm going to explain with a simple example here. the main thing I'm talking about is just the concept that representation shapes implementation.

5:56 so let's say you're implementing something like ticket master where you want to have you want seats to be in one of three states. You want them to either be open, sold, or held while somebody is in like the checkout flow. You can represent this with a somewhat naive representation on the left where we have a bull flag for held and a bull flag for sold. this, you know, this makes sense. This is what like >> a lot of programmers may do.

6:30 >> Like a freshman in college would 100% write that code. >> Yeah. >> no doubt >> you end up with this this crazy like state machine here on the left where you have you you have to have you have to handle all the possible error states because with this representation you can represent the state in which the seat is both held and sold at the same time. you have like Schrodinger's concert seat. >> Yeah. >> so if we want to go from held to some other state, we have to do this error handling to make sure we don't end up in an invalid state.

7:05 >> Whereas if you fold it into the data structure here and use an enum you can have open held and sold and the state transfer becomes much simpler. But what I found really interesting at the >> Go ahead. >> You basically create a exclusivity here. It cannot be held or sold. It's like it has to be one of it cannot be held and sold and it cannot be either. It has to be one of those three things.

7:31 >> And I'm sure everyone watching this probably like a so obvious. I get it. Whatever. But like go to I think this was the part that really made it really interesting to me. >> the point we're going to find out is that agents really love to screw this up, but that's only to bury the lead. >> >> I I think most of the people watching this would be like, yeah, like this is this is pretty obvious. You don't want to do this this representation. It's bad. You want to fold it into the type system and let it handle it for you.

7:56 But >> when you add like a new state, this is what I what I find really fascinating when everybody shared this, which is like >> adding a new state changes the amount of places where you have to touch the code. And this may also be obvious to many people, but think about like an agent writing this code. every single one of those like four places we have to add is now an area where like an agent might forget to add it >> and then you're screwed.

8:21 >> Yeah, it's it it grows exponentially. >> >> and just to be clear here, when you're returning none in these states, that's sort of your error case of like, hey, nothing nothing was changed because you've asked for something that is invalid. >> Yeah. >> Yeah. >> So you Yeah. But you have to handle that individually on each of these. Whereas here you can fold it into this wild card because it's handled by the type system. which cool >> makes it much less prone to error. Even human programmers like I'm sure like this is not a very realistic example like if somebody was doing this there's going to be a lot more flags than this >> like for like what is the state of your ticket order, right? Yeah. That that quickly grows to be your like god object with fields or whatever. Yeah, we're trying to avoid the god object here.

9:09 >> okay, this makes sense. And yeah, so I guess the take here is like if you if you have a bad representation, every time you want to change the code, it causes it to get even harder to change versus like code that as you change it, it stays the same amount or like gets easier to change over time >> and easier to understand. >> It's it's easier to It's often easier to change and it's often easier to understand.

9:34 >> cool. So, is this actually relevant for agents? I I hope if if you guys aren't tuning in for the first time, you're probably familiar with how this podcast feels about slop. slop is bad. And slop has been bad. Yeah, exactly. >> Slop is always >> slop has always been bad. We've always had like, as anyone who has played like a Bethesda game can probably attest to, we have had slop since before AI. There's been bad code and bad data structures for a long time. And we all know that slop leads to bugs. It's very it's very obvious. I I at least I hope it is. But do agents actually make these mistakes is is the real question. And to to get to the bottom of this, I devised an experiment where I had codeex GPT 5.6 soul on high. So like the Frontier model, Frontier Harness, build ticket master, but instead of like oneshotting the entire thing all at once, I had to build it one feature at a time just to emulate how teams already ship. So you know, it's it's not realistic to have all the requirements up front.

10:51 you're gonna I mean maybe for something like this simple it is but we we want to emulate like actual like an actual team where you write one feature and then you stop you hand it off to your co-orker and they write another feature and now this is all being done done with agents. So it's it's your your co-worker's agent and your agent. So it's actually really interesting dextra like different than slop codebench like what's fascinating here is like agents have no context about what the previous agent did. I guess that's is that exactly how soft code bench?

11:23 >> No, soft codebench is the same. It's kind of like cool like implement this feature and it has to go investigate the codebase. >> it deletes all the code up front. All the ticket information, everything. It only has commit history or something. >> I I don't know the exact specifics, but my my understanding is that it looks like this where it's like okay, you inherit the code and then it's like cool add this feature to the codebase and then it's like cool now add this feature to the codebase.

11:49 >> Yeah. I I don't think I even gave them commit history. >> Yeah. So, it's fascinating here. I think >> Oh, yeah. They don't commit either. No, there's no commit history for slot codebench. You just get the raw checkout. It's actually like people always ask like, why doesn't it use git? And it's like it's intentional. You just land in a raw codebase. >> Yeah. >> Yeah. >> Cool. It's like, all right, what's the conclusion? You ran the thing best model, best harness, you spent you ran every single one of these sample segments 200 times, and then you actually looked at the code.

12:18 >> So, like what did you find? So >> this is >> feature one. Sell seats. >> Yeah, sell seats. You know, a seat is a label that can be associated with a customer. That's also like it's their name usually. >> Pretty much all of the agents converged on like the obvious representation. at least for anybody who's taken a DSA class. >> It's crazy that it's not 100%. By the way, >> I I think some of them had some like strange >> Okay.

12:50 >> I I I recall looking at the three and it was it was odd. >> Oh, man. I wish we could see one. You don't have the code around anymore. >> I I do. I'd have to pull it up after though. >> maybe we'll do the next one. So, this one was easy. Basically, like almost a gimme. >> Yeah. So then let's say the front-end team wants you wants a way to list open seats and they want a printable seating chart which >> this kind of implies that you want like a a stable ordering to your seats.

13:20 >> Yep. >> So 40 like roughly 40% of agents made a manual index map where you have this >> this like extra ve that holds like the seat order. So this this ve of strings is is like representing the string is representing the same as the same thing as the >> the seat label. >> And I know for just for context here, I know not everyone here writes Russ all the time like we do. So obviously they're not going to they might not recognize this. The key the key problem statement here is that >> if you want an if you if you're famili like ordered dictionaries where you want to order list out dictionaries in order for the keys rather than like the insertion order or whether you want like random order for hash unordered maps and such in Rust there's a data structure called index map that oh not index map like well you'll see this right here that you can just use that's like very commonly used.

14:13 >> Well there is index map >> there. Yeah, but we don't want index map cuz we're not insertion order. >> But the fact that you're manually ordering the seats is just a bad >> it's a you would flag this like you don't want someone to write that code. >> It's somewhat defensible for if you wanted to keep insertion order because I didn't tell it necessarily >> but then use index map. >> Well, I I did tell it don't bring in external crates and notably index map is an external >> crate. Okay. But if I were to do it this way as a human, I would implement a strct called index map >> to do this for me. so I can test on that very end and not reimplement this every time.

14:53 >> Yeah. >> then some of them kept the exact same representation but they just sorted it at the very end which like that's also defensible but >> on request. >> Yeah. Yeah. not not optimal in my opinion. another like 5% for some reason added like an O of one like constant time lookup for the number of open seats which is just like >> flop. >> Like that's just like not what I asked for. I don't know what happened there.

15:24 Maybe some quirk with the prompt but whatever. >> Yeah. >> this is also bad. Like these are also bad because they they introduce it's this idea of like the the invalid states like this one has one invalid state this one has no possible invalid states. >> So we're we're introducing like the possibility if the implementation code is wrong to have an invalid state here where like let's say this is actually more than the amount of seats there are >> or this doesn't actually have every single label in it. and then 35% did actually do what I believe is optimal where they they used a B tree map which is like a like a self-balancing binary tree that that keeps these thing the the keys implicitly in order.

16:17 >> Yeah. >> So that does make the the lookup like login instead of the look >> instead of 01 or whatever, right? Yes. So if you are worried about performance there there are ways that you can you know you can cash this and such but >> orthogonal very trivial for the the purpose of >> you only have like how what's the biggest stadium selling seats. >> Yeah. You want to optimize for the biggest thousand seats >> like that's that's absolutely >> implied by the problem definition I would say.

16:50 >> But I agree. I knew concert concert venues for AI agents startup allows up to a billion seats for all the agents. >> True. Yeah. >> Just saying. >> We're going intergalactic. >> next we're adding save and restore. So, like let's say we want to be able to turn off like our ticket master service overnight and bring bring it back on. we need some way to serialize and deserialize it. All of the agents did the right thing here where they didn't change the data structure at all. But this does introduce an issue where or not necessarily an issue but it introduces coupling downstream where now we have a ton of downstream code for this serialization and deserialization logic that actually depends on the the structure or that depends on the data structure that we used. and with agents especially this I mean with humans too it makes it much harder to change things downstream because the moment you change the data structure you get like 500 errors elsewhere in the codebase >> and serialization.

17:59 >> Doesn't this depend on the ordering like I'm seeing where you we're pulling seat off of the loop >> but remember we already said we want it to be ordered up front. >> Yeah I implies ordering. I suspect the I suspect this example I pulled was from one of the ones that just ordered it on demand. >> Yeah. >> Okay. >> Yeah. next thing, final feature. customers are complaining that their seats get sniped when they're in checkout flow. We need a way to add five minute holds.

18:33 So 87.5% of the agents refactored it to what I believe to be the ideal representation. in this case that doesn't allow for any invalid state. So you have you're folding your sorting requirement into the the B tree map. so our seats are implicitly sorted. And then we use this this is a rust enum. Once again, sorry for using Rust, but it makes it very nice to to do these things. >> You're not sorry.

19:04 >> I'm not sorry. I'm really not >> sorry for using Rust. >> You can think of it for people that are devs, you can think of it as like a tagged union. >> Yeah. >> Is like the best way to think about this. >> Like the enum basically is able to hold like state and data at the same time. >> so it just makes life a little easier. >> Yeah, >> this is kind of nice. Like mo most >> I don't think TypeScript people call that a >> tag union say I don't think people call that a a tagged union that's a C thing I think they call it a discriminated discriminated union I think is the JSON schema >> word man freaking type I'm done I'm done with type desk >> >> all right then you're not allowed to use human layer anymore this human layer is all TypeScript >> almost all >> I'm sure Kyle will talk effect later.

19:54 It's fine. >> almost all agents over here almost all of these things like 12 and a half% did did kind of the wrong thing. They have like a lot of invalid save representations here. >> Yeah. Like there's three sources of truth in this data structure. >> Yeah. That's so >> not not optimal. The thing about this is that 12.5% of the time this is per feature. That's a lot. Like that compounds over features. you're going to end up with slop in your data structures if you don't watch them.

20:25 >> Like basically like yeah 12 and I >> and then I I hid something even with the right overarching shape. some of them added a second source of true truth for this time. I I don't know why this was I think it was some some difference between like the system time and >> like the time >> those weird time semantics >> for serialization. Very strange. Yeah, you have determinism time and wall time which is like human human readable.

20:55 >> Yeah. so this is not something you can really catch in CI. The tests were green the entire time. I had all of them write tests like they all passed. I guess there's no way to catch this. So the options to prevent it are either one know your know like your representation all your features ahead of time. And if you do this, the models are very good. Like I struggled to get them to produce incorrect output when I gave them all the features up front.

21:25 >> so what I'm hearing is skill issue. >> Yeah, skill issue. Rewrite your entire codebase every single time you add a feature. >> Yeah, >> you just need more tokens. 200 more dollars. Another chat GPT plan. >> yeah, obviously this is not feasible for a lot of projects or teams. >> yeah, you we I just took like a week and a half to rewrite our type system. Or you can review the data structures. And this is at least at this point with these models what I implore you to do to actually review the data structures because I think they're the most significant thing that affects downstream code the most. And if you can simplify those, you can simplify downstream code and be more confident in it, especially if you're not reading any downstream code.

22:09 so yeah, I hope I hope people agree with me now or at least some of you agree that all slop starts as bad data structures and I'll plug my info here at the end. >> That's funny. if you know, I'll I'll give people something slightly more interesting than this. >> I don't know, Dex. What are your thoughts? Agreed, disagreed? >> I mean, you know, my take on agents writing slop. and I think it's sort of like because slop compounds even a 12% like failure rate means that every time I run 10 prompts through my system if I ship 10 features in my software factory one out of those 10 is going to make my codebase worse in a way that compounds. That means after a 100 features my codebase will be like 40% slop. Like it compounds really really fast. And so I think like people say, "Oh, the models are good enough. They only make mistakes one in 10 times."

23:08 It's like that is not good enough to go lights off and let it like compound on its own in a trae because you have the trajectory of the agent session. >> But then you also have the trajectory of the codebase itself and that is subject to the same compounding rules that happen when you're 10% wrong, 10% wrong, 10% wrong. You very quickly become 40% wrong or 50% wrong. And at a certain point it's going to become unreoverable.

23:31 >> Yeah. 100%. >> Yeah. I suspect I suspect the 12.5% is a lower bound, especially because this was such a simple problem. >> Yeah. >> Yeah. >> I'd be surprised if it's not like it makes like bad data show decision like 20 20 plus 30 plus% of the time. >> Yeah. >> And this is by the way, not only is this in this is in Rust, it's making bad data structure choices. It's definitely making bad data structure choices in Python and TypeScript. Like it's not even a choice. Like it's like >> Yeah. And the agent will just do the convenient thing because that's what they're optimized for.

24:05 >> The train data literally just makes it optimized for convenience because honestly that's what humans do. They optimize for convenience >> and it will always be true in software. So it's interesting for those of you interested. This is the kinds of conversations that were going on at the AI unconference that we held. These are the kinds of topics that we'd like to go into. There's data behind them. There's meaning behind it. And we're like we're actively trying to share knowledge from experiments and workflows that are going on. So hopefully if you checked out last week's episode, this gave you a more in-depth detail and we'll try and bring on more speakers and stuff from the unconference to go and share their learnings with us if they're down to. I thought some of the talks were like fantastic. and I truly did learn a lot.

24:46 >> There's a lot of things to learn. There's a lot of smart people and we're all learning things along different trajectories and different dimensions. >> Yes. And and the thing I always come back to, it's always a skill issue. Apparently, either the agent skill or your skill, but it's always a skill issue. >> Or both. >> Or both. >> We can blame the agents. >> It's true. >> I do prefer to when possible. >> What's up, Kyle? How you doing?

25:10 >> Good. How are y'all doing? Good. I think we chatted about this a little bit. Kyle's has spent a ton of time working on our sync architecture. and when I say sync architecture, I will perhaps just like draw a tiny little thing which is like when you run human layer, there's three components. There's the API, there's the like client which is like desktop or web app or mobile and then there is like the what we call the demon or like the worker would this is the thing that runs your cloud sessions and your codec sessions.

25:48 And what happens is these two things are running on your workstation. and let's see, but every time you send a message to the agent, it flows through our API. and every time a message comes out of claude, it flows through our API. and by by our API being kind of our cloud, there's a couple different components. but we are able to do this very efficiently in terms of compute in memory and things like this.

26:18 We're able to do it very fast on the order of like I don't know I want to say like 50 milliseconds of delay between like claude outputting a JSON line and it shows up in your client. and I think we were going to chat about maybe like how might you accidentally invent this system from scratch if you were just going to build it from first principles and what would be hard and then like what is the actual like off-the-shelf thing like the same way you wouldn't you wouldn't build your own database from scratch, you probably shouldn't build your own sync candid from scratch. I don't know if that maps on to what you're thinking about, but I will pass it on to Kyle because he's been building in this space for probably a year now at this point.

27:01 >> Yeah. so I guess to start it's when we look at an architecture like that, it's probably worth mentioning like why we designed it that way in the first place. because that's a question we get a lot is like why don't you just have the desktop app and the native or like the desktop app and the Damon talking to each other directly using like a standard stream or some other like host local interprocess communication mechanism. do you want to pull the excel draw back up maybe Dex?

27:30 >> yeah sure I can share it while you're while you're drawing on it. The reason that we did this, and we didn't support this on day one, but we knew that what we wanted to do was to be able to have a bunch of different dammons and remote workers on different hosts, maybe like for a separate user's workstation. >> You guys kind of want to do this thing where like any machine can run and communicate with the central layer. And I saw this amazing tweet by you yesterday, Dexter's controlling codecs on my machine.

28:00 >> >> Yes. Yeah. so yeah, we we wanted we we believe that like the future of of software development is is heavily collaborative and so we care a lot about users being able to interact with other users coding agents to be able to interact with artifacts like plans and design documents created by other users and and and it doesn't make sense to like lock your work to a host in the same way that like it might for an IDE where you you have a local file system.

28:29 we just really wanted it to be able to just work whether you're on the web, whether you're on a mobile app, whether you're on a desktop app, and to be able to manage Damon's running on like sandboxes and VMs and the Mac Mini that you bought for your Cloudbot that you almost forgot about like and so this is this is what allows us to do that. and so there's there's a lot of like really interesting design considerations here. And I guess if you were going to like kind of design this from scratch, right? If you're like, "Well, okay, I need some central some type of central like system here. Let's move over here a little bit."

29:11 If you're like, "Oh, I have like my API whatever, right?" And then you have a bunch of different clients. Gosh dang it. and you're like, "Oh, I want updates like to and from each of these to be synced out to each other in real time, right?" what is what do you think is the first thing you would reach for? >> It's probably websockets, right? >> Yeah, websockets or like SSC before that of basically like, "Hey, I want to broadcast a message when something changes and tell these things to like refresh the data."

29:48 >> Exactly. or like a fire store or like have you guys used realtime databases like convex or something? >> Yeah, something like that. >> Fire store and convex are sync engines basically. >> Sync engine is a fancy way to say real time database. >> Got it. Okay, cool. >> Sorry, excuse my pleb brain that sadly has never worked on any sort of cloud product ever. >> It's a it's a better word honestly, but yes. >> Okay.

30:13 >> Well, it's also it's a spectrum because our sync engine is not a real time database. which we'll get into in a little bit. but our sync engine sits in front of our Postgress database, which is one of the reasons we like it. so like if you had a client that was just like sending a prompt, you know, or an agent sending some agent result, right? What you would have to do is you would then have to like have your API server kind of receive that on a websockets connection and then broadcast that out to all of the other clients.

30:43 the the first problem that you will run into if you do that is if you have that happen, right? You know, your agent sends a result and then another client joins and it opens a connection. It never got that message because it wasn't online, right? And so now you have to have like kind of a a websockets like room that has some type of state that you're adding to and then when a client connects you flush all of this the like kind of buffered state to them as well as any new messages, right? And so now you you need something that's like like this is kind of like what a Cloudflare like durable object is, right? You can have a bunch of websockets connections. It has a little SQLite database associated with it.

31:29 that that's like one way to do this. one of the problems here is like this is a highly stateful thing now and scaling websockets is is not pleasant because they are stateful longived connections and you need stateful storage in the memory of your process or you have to have like short-term ephemeral storage like in Reddus or Postgress for your your your buffer here. the next problem is you actually want to have all different kinds of data right that is going to and from each client. So like you might have something like you know in human layer we have tasks right which each task has a name and a bunch of configuration information associated with it. You have sessions you have I don't know >> I'm going to ask a really quick high level question really fast Kyle.

32:19 >> Yeah. What's the key part that's different here rather than building like a chat chat room? Like if I'm going to go build something like Slack, it feels very similar. What's the key difference here? >> Yeah. I would say one of the key differences is like there's so many different things that you have to sync. we we in kind of the sync world, I guess you would call them shapes. but it's like a a slice of data in the system that you want clients to know about. And for something like a a chat app, the shape that you care about syncing to every clients is just like the list of messages for that conversation, right? It's it's very simple.

33:05 >> it is it's relatively low churn, right? You're not like adding or removing or updating data in it very frequently. And >> all right the the right model I think is from what I'm hearing is something like if I wanted to build something Slack where like for example like one of the things in Slack is like you don't want to scroll up and load all the messages on day one because you could have like five years of history in there.

33:32 >> there's like some dynamic stuff. Is that is that the right as someone that's never worked on a sync engine? >> Yeah. >> Scrolling I think is separate. And Kyle, just you know, I I just added the OG like electric PC diagram off to the right there. >> >> Amazing. Yeah. >> Yeah. >> Is is it different because with your sync everything builds on the previous whereas like Slack messages, you're not really building on previous messages.

34:02 >> Slack messages do kind of function as an appendon log. and it is very easy to like get a fixed tail of that and then like load them in a pageionated manner. one of the things that is different about like the the way that users interact with data in human layer is like there is a very large amount of information that a client could could want to access. Like I have literally probably 100 plus sessions a week. Each session is associated with a task. It has ACL rules. It has it has tons of conversation events. And each of those conversation events, like the list of conversation events for a given session in some cases is is tens of megabytes because you have so many like files and diffs and >> thinking messages and stuff. And so it's a lot of data and there's not a like cleanly pageionated way to get that to the client very quickly. And so you you have like a large boundary of what the client might want to access, but you don't have a good way to like optimistically fetch for the client like what they're likely to access.

35:12 >> You can't predict intent effectively. So like because the data source that's actually really interesting. I think the now that you frame it that way, it actually makes a lot of sense. Like if you were to ask me like why don't I want to build my own Slack? It's some of the exact reasons that you said. We're just like I don't want to deal with file uploads and like how am I gonna like do that in a good way. It's hard. and like obviously for just a chat message anyone can v code a websocket based chat message.

35:37 >> Mhm. >> But building the whole thing and then making it mobile optimized for example like how do I make it so my phone doesn't like kill my data because I just downloaded everything up front. >> Right. >> That's actually really fascinating. Framing that way was like wow. Yeah. I would not have thought of that. That's great. I see why. So tell me what's the hardest part about this whole system? Is it desyncing of like loading and pulling data?

36:00 >> Yeah. So the the hardest the our the sync engine that we use actually has like a particularly like graceful solution to to the problem and there there's like lots of different solutions to this and you could like websockets is just a transport. you could actually use it for our >> for our system. But like >> conceptually what happens is like there is a we we can get a little more into the weeds on the architecture. So by like this API thing what we actually mean is we have we have a database right? we're going to make that green and it's a circle or not?

36:46 >> I I can I can clean up your stuff. You just tell me what to do. I'll I'll make the day I'll make it look like whatever you want. >> Yeah. Cool. Okay. So, we have a good database. we have we have a Postgress. so, it's Postgress. That's important. we have a our sync engine which is built on electric SQL. we have an authorization proxy. and then we'll get into like fast sync and like slow sync in a minute. But then we also have a what's called a durable stream server which is like backended by a file system.

37:34 >> Okay. And I'm going to ask you a a couple a couple and it's a SQLite based file system is what you're saying. >> well, so the the index is SQLite. The the data is not actually stored in S3 or something. Yeah, it's probably like S3 or something. >> No, it's it's literally just a it's a it's literally just a file system. >> like the ephemeral file system in the container. >> Yeah, >> I see. Can you spin up a new container for every single thing?

38:00 so no there is the the durable we can get more into that in a little bit but it it like the the SQLite index maintains like a list of like logs for streams and indexes them and knows how to find them. >> Before we get into any of the nitty-gritty here I have a couple of questions that just pop up from this which is how what is the hardest challenge here that someone should know about? So is the problem ingest of data and amount of bandwidth I'm processing. Is it the number of web requests I have to make are so many that I have to reduce the latency between them and I have to do consolidation and other things or is it along the side of the workload I do itself is intense on the backend side and therefore I have to worry about compute and latency of that like where are we spending the most of our brain power trying to solve >> for us I would say it's it's more about the right load actually which is really interesting. so like I said a given conversation for a coding agent in our case can can involve like tens of of megabytes of data and for a sync architecture like in our case we write that to a database like Postgress is is good for that it's a single writer like multiple reader configuration and like Postgress is fine it will handle that load but in order to get data out of the database quickly to the client like that's the key thing is the client I'd ask for a lot of data and so you have to have some way of quickly like storing and caching and getting data to the client in a way that's not like them constantly pulling your Postgress database every you know every second or every two seconds because that's that's kind of the other like naive approach here is do something like tanstack query where you're defining an API endpoint and then you're like constantly pulling it or like invalidating it when data on the client changes. Yeah, >> I was about to ask like you just said like if you have write heavy workloads but then you said the most important thing is to fetch data fast. I was like you can't have a single writer and get fast reads.

39:59 >> You just can't or like fast updated reads. >> Yeah. The the way I would think about this is like basically like you would want to let's say you're looking at a list of coding agent session like messages. So claude does a thing, Claude does a thing. They're being a new one comes out every half second or two seconds or whatever it is. And so the client wants to like on page load fetch all the ones that have happened so far and then subscribe to a hook basically.

40:24 And so what you could do is you could like subscribe to a websocket and just tell the client rather than sending it the data you could just say like hey by the way here's what changed or hey something changed go refetch and then you go refetch all the previous hundred events and the next two and so that's really really inefficient and then the logic gets into like okay how do I tell the client send it just which rows changed and then you basically need like an in-memory database of all of the hundreds of clients that are connected what they've already seen.

40:54 what they've subscribed to cuz they all some of them are like I want to subscribe to select star from this table where session ID equals this and another one is going to be subscribed to select star from this table where session ID equals this and then you have that for five or six different tables to hydrate all this data in your UI and you want to basically unconnect fetch everything and then the sync engine's job is to keep track of all the clients what they care about and then to when something gets changed Postgress to compute okay which clients do I need to send incremental updates down to >> right it's not like it's not so much like handling the volume of of rights is the hard problem it's more like the the number and volume of rights churns shapes very quickly and clients you have to you have to explain what churns shapes means >> so a a shape is like the thin slice of data that the client is asking for out of everything it could possibly ask for.

41:53 >> Yeah. >> And churn just means like that's changing constantly, right? So like Dexter said, when Cloud's running, it's sping an event out every like couple of sec. By the way, we also do token streaming, which is the reason we have that kind of separate faster sync system. but the the the clients are writing very quickly, which means shapes are turnurning, which means other clients need those updates. We want that to happen basically in real time. And so you need a way to detect as soon as data is changed, which is happening incredibly frequently. How do we rebuild kind of that the sync system has to rebuild the shape or incrementally update it and then push updates out to connected clients cuz clients unlike something like I don't know like GitHub or something which can know when it needs to invalidate something locally.

42:40 It's way harder to know, oh somebody touched this thing on some other server and I need I need to ask for that. But the server has to invalidate all the different clients. >> Well, what this reminds me of a lot is actually like video games designed. This is literally how like if you've ever built like a multiplayer video game, this is the exact architecture that you deal with. >> So, I'm curious now now the question comes up if that's the case, do you guys do something clever on the client side to basically make it really fast on the client side and do a lazy invalidation?

43:10 >> Yeah, client side prediction >> cuz you do client side prediction cuz it's running on the client host. So like if I can stream the tokens directly from claude onto my UI but then basically the server is authoritative and says if the server says something different I'll cancel it and like I'll re reupdate the UI. >> Yeah. So that that is how we do token streaming, right? Because token streaming you have like fast sync which is like oh I I want like an updated row of events in my conversation to sync out to connected clients in like 100 milliseconds say maybe like that's that's fast but it's not like it's not fast right? Yeah. token streaming you you know your claude might if you're streaming 60 tokens a second you're typically getting diffs from the the server center event stream from your inference server like you're getting another token update like >> sometimes tens of those a second >> how do you get that to all your connected clients without yeah >> they chunk it a little bit individual even then ideally they speed it up but yeah anyway go on >> yeah Yeah. So it makes sense. So you don't send out token streams to every client. You send out like you basically do another batch on top of token streams that get synced out to clients and then or do you not do that at all like no >> we don't want to >> we don't want to go through Postgress right because the sync engine is basically like write to Postgress and then everything else downstream is handled by the sync engine of like oh something changed in Postgress based on the write ahead log like the replication slot >> and then we we stream that down to clients. But if you want to skip the Postgress round trip and you don't want to put every incremental token as like an update or an insert into Postgress, we kind of skip that path and just go straight over the the durable stream.

44:56 >> Batching there though. Oh, so you don't do any batching. Well, it might be interesting like if >> we we do under the hood, right? So the way that >> Yeah. So the the way that the token streaming works is like so actually the better way to explain this is like the way that a normal write works first so we can like start with a normal write is like the client like the Damon or something. So that's not actually not our web UI. That's like our little Damon worker. says, you know, hey, I have like a thinking block for this conversation, right? So I'm going to put a thinking block.

45:34 And the API is like, oh, okay, cool. I'm just going to drop that in Postgress. And the way that the sync engine works is like Dexter kind of alluded to, it is tailing the right ahead log. So it's it's Postgress has what's called a ride ahead log for anyone who doesn't know much about Postgress internals. I'll try not to get too into it. >> Yeah, let's ignore that. Let's just assume that it works. >> Yeah, it's it's a log of of trans basically of of things from transactions. Like it's how replication works. It's how a bunch of other things work, but it's a list of the way that we're using logical replication. It's a list of insert, update, and delete operations. and the sync engine basically consumes that like a read replica would. So it's pretending to be a read replica as far as the database is concerned and like when it it knows how to it knows what like rows are included in what shapes and when a client asks for a shape and then it receives an update from Postgress about a transaction it says okay I need to notify the five clients that care about this shape and send them that update.

46:36 for token streaming where you have like you know a single thinking block might have like 10 or 20 or 30 like tokens >> chunks >> or like token chunks per block and it takes like 2 seconds. You don't want to write to the API write that to Postgress wait for it to go out on the replication log and then sync out to clients because that'll just massively increase your yeah it's it's too slow. it just massively loads your Postgress instance when you have that happening just because of the sheer volume of transactions. and so you don't want to do that. And so what the client actually does for thinking block tokens is it it's a it goes through the proxy to the durable stream server which uses a very similar wire protocol to electric and also to to Kafka actually. But it basically says, "Hey, I have like I have a like a thinking block which is like this, right? It's just a single token and some information about what stream it's for. And so all of the clients that are connected to that durable stream server are just receiving server event notifications as soon as things are written to this. It's HTTP cachable. It's very performant.

47:52 >> it will keep streams in memory. It will flush them to the file. >> It's just like a proxy. It's mostly like a proxy effectively what you're doing. and I see what you're doing. Makes sense. >> Yeah. So it it has a local like SQLite database and file system and so it's it's actually like as you're appending multiple events to it, it's writing it to the log on file system and then if a client connects >> the entire problem here.

48:15 >> >> how big is the durable stream server? cuz like that seems like a real-time connection machine and if you have a real-time connection machine you need CPUs so you can like do nice throughput on them. >> Yeah. >> how many how tiny of a machine or how big a machine can you run and how many concurrent connections can you on that machine? >> it's actually exceptionally fast because the like for example a single threaded bun JavaScript server can handle tens of thousands of of like concurrent connections for things which are not like dependent on upstream databases and stuff like that. So even with a single thread >> we we use SSE. So like the most compute intensive part of the connection management is the SSL handshake. And so like we leave a lot of like long live connections open.

49:06 >> Yeah. >> Got it. And it kind of just works. >> Yeah. >> And we're also it's a like what we're using it's actually a Caddy plugin. So it's written in Golang and runs on Caddy. So it is actually multi-threaded. But the slowest thing it has to do is talk to its file system when like a client hasn't asked or has asked for a stream that's not in memory and it has to go fetch it off disk and it already knows >> 100 ncond time.

49:30 It's incredibly fast. It's not buffering like too many things in memory except for like an initial stream fetch when it's fetching the stream. It's like hitting the SQLite database saying what is the file path for this stream? reading the stream, sending it to the client, and then notifying >> even my stream tokens are actually not going directly from my Damon to my UI. It's going my Damon to the cloud to my UI. >> Exactly. And that's why anybody else on your team can watch the same thinking streaming in real time.

49:57 >> So now I have a question. Does that mean that I get a lag when I use it by default? Like I'm probably seeing like like anywhere from like a 30 millisecond to a 120 mill like >> I got to pay the ping cost. >> Yeah. Yeah. Yeah. I mean, so you do but >> like >> I guess more offset than it is anything else, but >> Mhm. >> Okay. >> Yeah. Like the the the jitter is quite low. It's just like you are seeing whatever your coding agent is doing like a few tens or hundreds of milliseconds after it's it's actually done it, which is like not that important in the grand scheme of things because coding agents are quite slow, right?

50:36 >> Inference is slow. >> Yeah, inference is really slow. and and so like it turns out you just actually don't care that much. most people don't even notice like you you would if you were using human layer like you would just not be aware. >> One interesting thing that you guys could do that wouldn't be that hard is you could just draw make the UI happen directly on the Damon and just do invalidation on this end >> for the primary client. You can make it so that you're invalidating off a network >> just to >> huh >> just to optimize all the way. Why? Well, like >> yeah, >> like for example, like cancellation commands for example, I've noticed like take a while to register through and whenever you cancel and part of that is probably because you're doing the round trip and like what's nice when I use cloud code is like I just hit escape and it cancels like boom. So when I when I experience that lag, I'm experiencing that lag as an end user.

51:31 And what you really want to make sure is that the primary you're right that you want to make sure readers are good, but you want to make sure like cl like video games do this all the time. So like I'll give you a really cool example in like first person shooter games when you have firstperson shooter games obviously a player can be experiencing lag. But when you're sniping someone and you see that you made a headset, a headsh shot in your UI, what ends up happening is even if the cloud sync engine says technically you would have missed because of lag or whatever, if you perceived it as you killed the headshot, you get that head shot.

52:03 >> There's a really good blog post about this. >> Yeah, it's an you can find Yeah. >> And like the reason is like the p it's such a bad experience for the person to feel like they missed it because of lag. And the person that's running and doesn't know doesn't care. like they're basically like they died anyway. >> You don't even need to like it's it's almost a UX thing. You don't even need to even like actually send it locally.

52:24 You can just make it look like it >> exactly >> canled. Yeah. >> So like that's kind of that's kind of why I was suggesting this is like it would change the DX of the person interacting with the person a lot >> and make the system a lot better. So, so the the actual answer here is like the thing which you are perceiving as slow cancellation is us waiting for the Damon to finish flushing the events that it has written like out to the API before we do that. one of the cool things that we do have in our sync system is optimistic mutations and the reason that that feels slow to you is because we're not doing one right there. We could we just like have not done that.

53:03 >> which is entirely like cancelled until the demon actually like is acknowledges like yes I've stopped the claw session >> but can't the demon isn't that all local so the demon can boot everything locally right there's no >> it it can be IPC but it's not necessarily right if you're running it on a separate machine it's not and so then you need like two separate paths for writing an IPC and the demon has to know whether it's expected to be talking to a native app or not And and so we've opted to minimize the coupling, right?

53:36 >> We could do this as an optimization. It would be very special. It would >> it would be a very specialized optimization that we could do. >> Specifically for cancellation be dope because like that's the part that like if I'm canceling I want to cancel now. >> I I think >> I guess if we had 99% of this is making the UX like it just needs to look like it canceled instantly. >> Yeah, I agree with that too. If you made me feel like it canceled instantly and it didn't, I'd also be happy. You you would never know.

54:02 >> I literally wouldn't know if you're burning tokens. >> Make the interrupting state not display to the client and then only like reverse it if something broke. >> Yeah. >> Yeah. I mean at some point like I guess like for this kind of thing what you're really competing against is like the native app that someone's using and if your app feels awesome then it's worth it. And the collaboration is a huge huge value prop. I think obviously like the fact that Avery can like pull up the same session as me and share it is like a great value prop.

54:35 >> The fact that I can take that session like pull it up on my phone while I leave. Great value prop. I was just like, oh, if if I had to think about it purely from a system design perspective, I would go study a lot of the video game art and be like, how do video games I would use UDP, by the way. I'd stop using TCP. I'd just like build a UDP session around this and you'd get like massive stress. But you you lose the ordering though.

54:58 >> You can still do a lot you have to every time you use UDP, you reinvent TCP on top of it. >> Depends. >> Yeah. This is the thing that happens. This is why >> this is why almost nothing uses UDP >> except video games. >> Well, they use UDP and then they re they invent minimal TCP >> on top of it. >> Yes. >> But they build the whole they build threearters of the TCP envelope inside a UDP packet.

55:19 >> Yeah. But that three quarters is why it's fast. >> Okay. >> okay. Like, >> this is fun, guys. >> I mean, a lot of stuff in like video game stuff doesn't need like exact ordering, >> I think. Well, one one thing I'm wondering about is are the are we are we largely sending like delta encoded like are we sending like diffs and these diffs like depend on each other or is are we getting snapshots every time when there's no dependencies from the previous history.

55:45 >> So, when we're are you asking about when we're doing a riot or when like data is getting pushed to a client? like when data is getting pushed to a client. >> Yeah. So this is yeah that's a that's a great question. the way that I guess both of these systems work actually right the fast fast and then like the slow one which goes through Postgress work is they are both aware of the client's state. So when a client connects it says hey I need this shape >> and it just says my cursor is like negative one right >> and if the sync system already has that shape in memory because another client's listening to it sends it to the client and then the client like updates its cursor with the last received event and then makes another request and keeps that connection open until there's another update available right it's how long pulling works little different from SSSE >> and for durable streams we do actually use SSE because you just get you you just get faster updates with a single long live connection that you can write events to.

56:46 >> And it's also cachable in a way that long pulling is not, right? Because long pulling is a bunch of like >> repeated requests, SSC is just a single request that you're writing data to over time. And so if you've already written a bunch of stuff, if you configure your proxy and your cache and everything properly, you can just serve that off of a cache without ever hitting the the upstream server. and so you need all kinds of cloud infrastructure and request collapsing and and like that. But that is that is possible.

57:16 >> and so the client if if the if the sync server right either the fast one or the slow one doesn't have the data that the client's asking for. It has to go and do a snapshot right the for the fast sync server that means a SQL light lookup and then pulling a log off a file system and sending it to the client. For the slow one, it means running a Postgress query to go get the current state of a of like whatever shape the client has asked for as well as whatever the the checkpoint was in the the write ahead log. And then as updates are received off the write ahead log from that checkpoint forwards, then it also includes those updates when the client reconnects. And every time the client gets a chunk of data, it's has like the last event ID, I think, as its cursor for the next request. And then the server knows, oh, is there data since this event that we can send to the client or do we just re leave the request open until there's data to send to the client?

58:11 >> Yeah. >> So, it's highly >> parallels with video. Yeah. The parallels with video games here are crazy because this is like >> exactly how a lockstep real-time strategy game, at least back in the day, would do this. like a lot of games you couldn't connect midway through a session because you didn't have the history and they didn't, you know, snapshotting that much data at that time just didn't work. >> Yeah. >> Yeah. That's why they have bandwidth is cheaper. Most games are like 5 to 10 minutes.