Why Performance Engineering Breaks AI Coding Agents

0:00 Because again, performance engineering, nancond level performance. I make one mistake, everything is and I just wasted my last three days. I need to micromanage every bite that I'm passing through the system. Do less work, use less memory. And thing number three, use caches better. >> And the rest is just how much do you understand the system that you're working on and how it works under the hood. >> Today's episode was a really fun one. We're going to get a little chat about the performance optimization loop. It really boils down to hypothesis testing.

0:28 we go from like how do you go from like building a performance loop, how do you define performance optimization and then what are the philosophies that your organization stuff might have to adopt in order to make this work. We take a look at a couple of different examples. We look at some chatlog examples. We look at what telemetry can help you drive and then we also look at some chatlog examples of how we might do some interesting work on here. But I think the most interesting conversation today was actually led by both Kai and Tamas where we went ahead and actually discussed exactly how we do less work and what are the philosophies behind it, how it sometimes relates to security and how do you build or culture around this.

1:03 Let's get started. Hey everyone, I'm Vibov and this is AI that works. AI that works is where we try and show real working systems where AI pipelines hopefully help get past the hype and show how production systems drive value for end users. I'm one of the co-founders over at Boundary. Normally I have my co-host Dexter with me over at Human Layer. He's sadly out in Europe prancing around not doing important stuff. So instead, we've got an amazing and probably a more exciting guest on the show today. Kai, you want to give a quick background?

1:35 >> Yeah. I've been here at Boundary for 7 months or so. >> Yeah. >> Mostly work on compiler and runtime stuff. I yeah I don't know. Cool. >> Yeah. And then we'll get to we'll get to going and talking about we'll have one more person hopping on a special guest halfway through. So hopefully you guys will get a bit. But today's episode is actually one that I'm particularly really excited about. And the specific thing I'm really excited about is actually to talk about how do you get AI to solve really really really hard problems. One such hard problem that I think is a really good useful exercise and maybe be a good proxy for other people trying to get AI to push the limits especially with like reliability systems is performance engineering. Now over at Boundary we do a lot of that where and some of the stuff that we do operates in nancond level time and typically you'd probably want to put a human on that type of code because well I mean how do you how do you get nanocond level precision if code is churning all the time? You can't because it's changing all the time.

2:39 I think today's conversation is going to be a little bit more free form, but I'll try and share some chat examples. I know Kai has some opinions on this end, which will be kind of fun to go through on our team. He's been doing some work for like our GC and our core memory model and a bunch of other runtime level semantics and the type checker. And then I've been doing more work more recently on the actual VM side and making it really fast. So as we go through, if any of you have any questions about exactly how you might apply this to your real world scenario, I would think of it as more so imagine you have a system that is extremely rigid design constraints that are very expensive to test. How do you leverage AI and agentic coding in that design constraint system is the general topic for today.

3:21 Now I we we've been doing a little bit of work specifically on the infrastructure side of this that's actually I think a little bit more important than the actual nitty-gritty of how you do performance engineering. And one of the core things that I want to talk about let's go to scale draw. we're not going to talk too much about the GC stack and everything, but I think what we will talk about is actually on the order of I want you to think about how you have to do performance engineering. Your workflow often starts with look at code and Tai tell me if you agree or disagree. Think of better idea or I guess see >> is it slow? Huh?

4:07 >> See what can go wrong. >> Is it slow? Think of better idea. Yeah. Then after as soon as you look at the code, you think and see if it's slow. Then you Well, I guess this is probably the word. Sorry. and then after you think of a better idea, then you like implement better idea. And I'll put better in quotes. And then you kind of just like at this point you're kind of back to where you started where you're like, "Okay, now is it slow?" And you kind of run this loop over and over again. Is that kind of how you >> I think in some aspects you would want to do this. A big part of it is just avoiding pessimization throughout the entire thing.

4:51 >> Avoiding what? >> Pessimization like the opposite of optimization throughout the entire process. So you're like at design time make sure you haven't made it effectively slower by design. >> and obviously for the hot VM loop you need to get that nancond level precision you kind of have to go through that. But I think throughout the entire process you really need to be >> yeah so there's a higher level design construct that if you know performance is a constraint you kind of need to think about it all the time. But there's also like a tactile moment that we want to think about. and I think you're right that we should maybe it's interesting because there might be like we might want to design loop versus like implementation loop.

5:35 >> Yeah, >> cuz like you're right that during design you want to think about performance and we'll share that a little bit later today when we show the chat gbtops about how we're just thinking about it without writing any code. But then there's also an implementation loop side of it. Now, one of the key things that you're probably noticing on here is that there's this measurement stage that is very very very important. What does the measurement stage mean? That means you are bottleneck on how fast you can measure your system. So, for example, if measuring your system it requires like nancond level precision, well then not only you're bottlenecked on how you measure your system, there's another bottleneck. your system cannot be doing any other work at that time because your measurements will be too diverse. So now you need a different machine that has some measurement capabilities that is somewhat stable to go do this in some way. So the amount of parallelization that you can do at that time is highly limited. You can't run like we we run a Rust code base. I can't run five workspace at the same time and do performance optimization in one of them.

6:35 It will make wrong decisions. Yeah, I know in our our test suite, there's a few tests that are performance or timing dependent with just like multi-threading and stuff to test that and they consistently fail whenever I'm running like three work trees at the same time. >> Yeah. >> Yeah. >> How do you get around that? Even beyond that, just like if you're running too much Chrome, Chrome is a resource hungry application and it's gonna it's going to mess up your measurements.

7:08 >> Yeah, there's this really interesting blog post going around right now or this image going around right now if you haven't seen it. It's like the bottleneck has basically moved downstream. And the thing is like if I were to think about what the most expensive time was beforehand when I was writing doing any performance engineering before AI literally the most expensive part was thinking of a better idea and implementing a better idea. So therefore it didn't matter. And once agents write all the code my only my own neck is how much can I measure at any given time. So like we're kind of just stuck. So like what I do for example if I do performance measurements we don't have remote machines that we can run on yet sadly. Maybe we should and we should do that. But what I end up doing is I just only do like re like non-coding tasks while I do this and I only do like design docs while I'm doing any performance measurements on my machine to guarantee it's stable. Always have the wire plugged in and such. But that's just a mechanical thing that I would recommend everyone think about if you're caring about precision for like timing level sensitive. If you're thinking about like like I would say like less than 100 microcond precision, you kind of just need to set up your CI system to go do that.

8:12 But I think the more interesting thing is how do you do the second part? How do you do these three parts leveraging AI? Because once you do the mechanical fix and you have this better, how do you actually look at the code, think of a better idea and implement the better idea? And implementation is tricky in performance. At least in my experience because there's no guarantee that the idea you thought of is actually going to be maintained by the coding agent that implements it cuz it will mess it up somehow.

8:39 How often have you seen that happen? basically every time. at least for the first try or two. yeah, I mean I especially it really is like domain specific, right? Cuz there's a lot of things that are intuitive or that are within the inference capability more directly or like I guess the training data, right? Yeah. of like this is how what what patterns it knows what to do and this is the kinds of systems that it has lots of pre-training data on and then obviously it can try and do some research but the context window is only so big and you can't have all the information in the world in there. So, and especially with like a lot of optimization is the things you would think intuitively actually speed it up.

9:34 Sometimes just don't >> because there's some other constraint or invariant that leads to >> Oh, yeah. Some in a second. >> That is one thing I think models are pretty good at is rel me or repeatedly measuring things. Although sometimes on my computer they measure things and then they're like, "Oh, wow. It just randomly became slower." And then it's like, "Yeah, that's because the other work tree just started a test run." >> Yeah, exactly. >> But you know, the getting them to really understand the core question is bit challenging because they'll go off on tangents about something that is not the main point.

10:21 and before we get more into detail, welcome to Tomas. Tomas is the right way to say it, >> you want to give a not >> you want to give a quick intro? >> Sure. hi guys. I'm I'm Tomas. I, I'm currently building a startup called Perf Soup, which is, going to help you keep your software stay fast as things change faster and faster around it, everything, right? and yeah, I've been doing performance engineering for about like 15 years since I started my open source load testing tool called Vetta. That was kind of the the kickoff for my interest in the area and I've been working on on this sort of problems for for a long time. It's kind of my my wheelhouse.

11:11 >> I'm super excited. When Dexter mentioned what you were working on and some of the stuff you did, I was like this is going to be such a fun conversation that we talk about. I think we were just talking about the general performance loop for a second about how you have to do there's like an implementation phase and Kai mentioned like you're right that like hey during design phase you just have to consider performance or else you will not be able to catch up to all the debt that you incur over time.

11:32 >> Yeah. But but if you but if you need to catch up you can use per loops. >> We'll get we'll talk about that in a second. and then I think there's a few other folks talking about like the general stuff in chat about some things that are Let me answer see if there's any questions really fast before I continue. Is this where like blast radius analysis come in for work definition? So blast radius is one element of it, but I think the thing that I have found in our core loop is blast radius doesn't really help because the thing that ends up I mean it kind of does but everything is so localized already in our core loop because usually once you're doing a performance sensitive perform like nancond level precision you're no longer trying to hit like you're just not reaching out to a bunch of code outside of localized things. Like usually performance boils down to three things in my opinion. It's thing number one is do less work.

12:30 That's a great way to make your code faster. Just do less work. Thing number two is use less memory. And that's kind of like do less work but this is I would say more like about how many instructions are you operating and this is how much space you're using to represent your work. And thing number three is use caches better. If you do those three things, you usually have a pretty performant system as long as you're optimizing on there.

12:59 There's special things like understand your hardware. But I've generally found that most people don't run on single hardware systems. So therefore, optimizing for hardware is just less common. But usually these three things are like generally applicable. Tomas, would you say this is generally how you think about performance engineering? Are there is there a fourth category that you bundle stuff into? >> Well, no. I I don't think so. I think I think you kind of made a hierarchy there. where like if you squint like use less memory and use cache is better is is all a version of do less work.

13:36 >> Yeah. >> You know, >> I can make it more specific. >> The is do less work. It's like just do >> Yeah. I'll I'll make it I'll make it slightly better. >> But I do agree. >> Do less work and do more of the work at the same time. >> Yeah, exactly. Well, >> those are the two things. >> That's a view, you know, they go at a specific clock rate. and they go up and down sometimes. But yeah, if you do less work, then they just eat more at every single instance of of time that it lasts.

14:07 >> I guess a better way to put this is really it's just what all the Neolabs are trying to do. Use less electricity. Yeah, >> that's actually what it's really about. >> You can really like, you know, peel the onion until you go to the poor, you know. Yeah, >> exactly. Use less electricity. But I I think the example that I want to share today as I go down this and like I'd love for you guys to roast this example actually as I go over here is some code that I've been working on more recently.

14:35 And I wanted to show as I did this code when I pull this up how I've been approaching performance work and how we've been thinking about it for a while because I think I have all the stuff on here. Let me scroll to the very top. Sorry, it's going to load. This is a very long chat thread. >> Yeah, I love that red minus 43,000. >> Yeah, this it do less work. >> Do less work. >> proof is in the pudding.

15:06 >> let me scroll up really fast. So a large part of when I was doing this is okay while I scroll I'll generally describe what I was doing. I was taking our whole VM loop. Our VM loop for a while was faster than Python. Then over time it slowly became slower and slower than Python for for the BAML language. And eventually I was like that's just not okay. How do we get to being faster than Python again? So then I just started deleting all the code that we don't need and started redesigning some of the architecture. So it boiled down to if I go back to that loop that we were drawing earlier. I detected it was slow. I looked at the code and I realized that we probably need to rework our core like telemetry system to make it faster because one of BAML's value prop is we want to automatically telemetize everything.

15:50 Telemetriizing everything has a cost. That means you have to read a clock. For every function call you have to read a clock. You have to save eight bytes of data that no other language saves for every single function call. That's that's more instruction counts. That's more memory. That's you just can't use caches better at some point because your stuff is just bigger. So when we looked at this code, one of the things that we ran into is we were just slower. And then when I looked at the code and I started looking at this, I was like, I have a couple ideas of how we can make this faster. And then we I spent about like maybe a day or two purely in the design world of designing this. And a large part of designing this was actually focusing on use less memory. I couldn't guarantee use less CPU instructions because of the way it worked. So I actually just said let's just focus on use less memory. Let's make the amount of data we pass through go down from let's say on average like 60 bytes of data down to like 32 bytes or 24 bytes. So we modeled that. And once I modeled that, I was like, "Okay, I want to implement this." But as you can imagine, going into the core VM and changing this codebase is a very fundamental change. Now, models, this is my opinion, and I'd love to know both of y'all's opinion. I don't personally think models are very, very good at incremental changes and maintaining two states of the system at the same time.

17:10 What do you guys think? >> I definitely agree. >> What do you mean by ahead? Like let's say I have a telemetry system that works today and I want to implement a better telemetry system. If I just tell it to say here's a telemetry system today, implement a better telemetry system, it will to use the it will not do it well is the best way I'd put it. It will definitely leave >> legacy. just pray, you know, make it good, you know, and and we don't define what what good is, you know, then then you're going to have a bad time probably. Like >> even with steering, even with steering, it's going to leave an extra data value there it doesn't need. It's going to leave an extra field around that it doesn't need to carry. It's going to start doing extra additions that it doesn't need to do.

17:52 >> Yeah. >> Yeah. requires requires a lot of >> attention that you need to put into the babysitting and the steering, right? >> Yeah. >> Or a lot of spend and and a lot of automated verification of all the things that you care about that then you know pressures the model into the right outcome. >> When I when I thought about this, I actually have a different approach that I tried. I basically told the model, get rid of all the tracing code, existing tracing code in the codebase. And this I did automatic verification loops that would just delete, delete, delete, and it forced it to delete every single line of code.

18:30 >> Very good. >> And I found that once I was able to delete it, I have much more confidence that the next loop that I start that tries to implement this much smaller data structure is way more likely to be pure. >> Is that a strategy that you guys have tried? Yeah, I think I've seen a lot that when I try to have it fix some system, whether it's performance or just like maybe there's a bad model that like allows invariance to be violated in the representation instead of having the language like Rust, we use Rust for the VM, right? Or for compiler and everything. And if you want it to have the best representation that basically prevents you from violating your invariance and so only valid states are representable, it often will just be like, yeah, we're just going to add this other thing because it existed before.

19:28 And even if it does like does it right the first five functions eventually it's like oh this other thing exists over here even though that's the old bad version that we're trying to delete and so I'm going to copy what it did over there but like no that the entire point is to delete to delete that. >> How how do you how do you find this ch changed over time like over the last six months? >> I think this one it specifically has not really gotten better. It's I agree >> somewhat of fundamental just like the models are I mean it might even just be the attention model itself but like trying to optimize for oh trying to optimize for like getting results is like it's a it's a general problem with RL and stuff, right? Is it's hard to in reward sparse like training runs. It's hard to prioritize the long run win over the we fix this bug now.

20:46 >> Yeah. And I think like Dexter has done a lot of talk on this on like slot code bench and everything where like we're just like at least empirically I see this as well where like if you give a model an open-ended problem even the same model is going to make different data structure decisions every single time and therefore if it has legacy code that it's being prior by >> it's almost always going to like be prior accidentally at some point in the context window by the legacy code because it'll just like forget get for a blip that oh we're trying to delete this thing >> and this thing is this thing is like evil code or like maybe you forgot in your context window to say that this part of the codebase is also slop.

21:28 >> Yeah, I think even when you do say that it eventually will just be like >> nah no cuz you know the the overall majority of the context window is that slop code that and it's like yeah okay we're just going to do that it's on average this is what we're doing. I mean I don't know if that's actually why but it seems to do that. >> yeah my experience that that happens and and >> the the way I found I need to handle that is is that I I can't just trust like every single agent session to to maintain the garden for me, right?

22:05 Like I need to sometimes go in and and do like a a big day of gardening, you know, like >> or or a big week of gardening. Oh yeah. I mean it depends on how much you you punt it, right? >> but but yeah, >> I I do see that. I I do see that even if you have very structured adversarial review and and >> llinters and all sorts of like deterministic checks that you try to that you try to encode your the invariance that you want out of your system. those help absolutely they help that like your code would be you have a much much worse time if you didn't do those things but they're not perfect right and so like >> and so they they still leave a trail of >> things that you would do differently if you be doing everything by hand like let's say right like >> yeah yeah I and I don't think the goal is to do everything by hand because also again if I if we go back to like that diagram we were drawing earlier doing this by hand Like you can make a ton of money pre AAI doing this by hand cuz the hardest part was this and you're literally paid for like ideas per minute merged in. And the thing is like now with AI like the bottleneck is no longer this as much. It's more like there's this extra thing of like did it do what you think it did?

23:31 >> Yeah. Verification. Yeah. Getting trust, right? Like >> Yeah. And like you kind of want to do verification almost like after after you measure it in some ways because you kind of the measurement is so cheap. >> Yes. >> and the verification happens like after the fact is kind of how I think about this. >> But I think it's like a there there needs to be a better systematic approach here. And like actually I think there's a really interesting question like when do you guys do gardening?

24:01 I can tell you when I do gardening for this problem that I have been working on, I basically spent all the time designing the entire feature up front without any work at all like writing zero lines of code. Then I spent a whole amount of workload deleting all the old code and saying I just want to delete all this code, delete everything, then measure the performance, then double check that you deleted everything and then triple check and then quadruple check. And then I trusted it that it probably deleted everything at that point. And it maybe didn't maybe did but I I think on a fourth check the models today are pretty good to be like did it delete code confidently that it doesn't need and by verifying what like our VM is pretty verified at least to some degree for some definition verified. So I believe that it did it correctly but then while I was deleting things then I said okay now make the core system faster without adding traync. And that I believe again is highly verifiable loops. I can actually measure it. I can go do things on it. And I can also do things like for example different types of measurements exist.

25:03 One of the measurements that I had to do very con constantly if I can find it is actually I think let's see if I have I might actually have a cloud artifact that I can show RT. I think and this is very much speculation but I imagine that sort of workflow might also have benefits for the context management because if you're entire recent context history is delete this type of code then it's much easier for the model to focus in on just that. and then if you follow that up with optimize this core system then it's becomes much easier. If it's just doing this task itself and doesn't have to focus on broader or higher level thinking then it doesn't get distracted as much. I mean obviously that's speculation and results will vary but there is some degree of correlation that I've seen with that sort of thinking even if maybe that's not actually how the thing works under the hood.

26:10 >> Yeah. So the thing I ended up doing while I was doing this was literally just have it build a bunch of diagrams that were just outlining the core architecture. And once I defined the core architecture, I was like, "Oh, how many ops can I do each of these in individually and just measure CPU instructions that I was writing into the codebase after I wrote the memory model?" Then I measured how many CPU instructions can happen, what race conditions can happen, and what scenario. And eventually I was like, "Oh, this thing is fundamentally always going to be a bottleneck because even the best algorithms that the best research papers have are always going to be like too many nanconds that the back pressure here will build up and then we'll basically cause too much memory usage up front because the core system is working. So then we actually have to just stop and explore a new idea. And we explored a new idea and did the same exact thing where we wrote every single thing as an independent script without writing any code in the existing codebase just like measuring op codes and measuring cache performance and measuring things in a totally standalone system. And I think eventually I got this down to like sub 10 nconds in a standalone benchmark. So then we can say okay we can make our telemetry system in theory 10 nconds. Now we have to merge it with the runtime and see how fast we can go do this.

27:20 And I think that's at least the architecture approach I have. And once this works, then I can delete all of the code that I have ever outstanding. And then once I delete all the code, I tell it to literally delete all the code, do nothing else. Then I start a new context window and I say now make this faster. And then I sort of once I've made the core system faster without any existing things, then I delete all of that. Then I make a new context window where I start actually doing the new work over here. And Tomas, I'm curious when you guys do your performance loops, is this what kind of loops do you kind of think of? Are you doing first principles? Are you saying this function needs to be made faster and do you rethink it or do you take the approach of let's just incrementally make this function faster?

28:03 >> Yeah. So I want to say a bunch about that. but before that I want to answer that chat question like how much you guys do gardening. have you heard like the Nvidia co with his phrase saying like as much as needed as little as possible? >> Yeah. So that that's kind of like as a as like a startup founder, you know, it's like that that is kind of my stance, you know, it's like reach a point where the pain justifies the return on investment of like doing the gardening, you know, like and so it's it's a it's a very dynamic situation on like slop expansion and slope contraction, you know, over time. It's if you think of cash flow in a company is like you have both sides going as as as time progresses where >> so how often what percentage of time would you say is gardening time for your team as a whole?

29:07 >> I I would say probably about like 25% of of the time. >> So like more than one day a week. >> Yeah. And and and I think that like we spent a lot a lot more time doing things with more human attention in the inception of the system and the architecture, right? and we put a lot of effort in in like trying to make the system >> be as agent friendly as possible like right to to try honestly as a as an experiment. Like I say, we're going to put our all of our eggs on this. Let's see how it go how it goes, right? But >> the the idea was that we >> so we couldn't we couldn't win or have a even chance of winning at this s like if if we didn't leverage agents like to the to the maximum right at this point in time.

30:05 >> so it's like this, right? It's like spend a lot of time at the design loop phase basically like as much as you can up front to make sure that like your infrastructure is set up correctly, that your directory names are correct that like small small wins like that. >> Yeah. Yeah. So, so I I I like to to think about it as like control the ideas >> as like like a madman. Like control the ideas as like a madman.

30:30 my aim is is is to be able to delegate the execution of those ideas as much as possible and and more of it over time, right? >> yeah, >> but but the the way to control those ideas, right, it it keeps changing, right? And it keeps yeah, it it just keeps changing. >> How much time do you think we spend on gardening as a team? I'm curious to see the numbers. I don't know on an individual level because I think well hopefully we mostly garden as we go which lets us avoid building on top of things that need to get replaced under it. cuz there have been a few instances where we have built things and then we had to and then like we built stuff on top of that and then if the foundation needs to get fixed then now you need to rebuild everything on top of it as well. but I think we've gotten a lot better at making sure that we reliably get it as we go. And obviously that makes it a bit harder to measure how much of our time we have to spend on that because >> you know, it's kind of built into the whole process. I think maybe >> it's like if you think about the language, there's like different layers to it. There's like syntax, there's like runtime, and then there's like our tooling effectively is kind of how you can think about this. And we actually really spend most of our time making sure that the runtime is correct. And I guess even more lower than this is like design.

32:03 >> Yeah. >> Like like the concept is kind of what I designed this as. Like when we designed threading, we spent forever just debating the concept. We probably spent more man hours debating the concept than we did actually implementing it. For sure. Yeah. And and trade-off, right? Trade-off, right? Like because >> yeah, >> you can get from zero to 80% of the implementation like this, right? Like and and so >> controlling the design is the work primarily.

32:31 >> Yeah. And I think the way I think about this is like what depends on the thing that you have to be right. >> So like the more things almost nothing depends on the syntax. Like yes, user code depends, but like if I'm completely honest, in the modern era, obviously you don't want to break user code, but it's much easier to not break syntax than it is to not break the runtime. >> If you made a fundamental wrong move in the runtime, you're going to break the syntax, too.

32:57 >> Like it's kind of like everything flows from a downstream. >> And and how how many like layers of of quality control do we have? I mean, sure, you have your core developer loop like in your computer, your design process. your test and and so on, right? But like for instance, do you have some sort of like pre-production environment where you like let things soak? you like use it a lot, you dog foods a ton, you have agents do do their own QA in in a pre-production environment.

33:27 >> I'll show you something live really fast. Let's see. Let me open it. What we do, what we actually just put this up right now is this thing. So every single issue that a user files from happen just gets really quickly enriched by other agents trying to write code and starting to see what the issues are. And so we I would say like for a language I mean we expect engineers to own the product end to end but like really what we end up doing is a lot of like downstream fixes because bugs like in a programming language you're going to have bugs. Python has bugs 25 years later. Rust has bugs. The best languages all have bugs. We're not going to be a bug-free system.

34:06 >> Yeah. >> So, it's more about how do you reduce fatal bugs and how do you make it possible to quickly address bugs that come in? >> Yeah. >> After the fact. >> So, I think that's just like an agentic loop that you can kind of it's like a standard like factory level system you kind of have to set up. And like we're still building ours out. I wouldn't say ours is nearly complete enough. But I think the most interesting thing that I have found is like in the mechanical part of actually driving an agent. Let's say we've done the design loop. Let's say we've done the implementation. We've found an idea.

34:35 We're implementing it along the way. I actually am really curious about both of y'all's workflow here as we're doing this. And one of the most interesting tools that I found is I moved away from the terminal more recently. As I started doing more technical work, I use the terminal less and less. And the reason and I found that really surprising, but the reason is twofold. two, you can see very common that I'm using annotations, but the other thing that I'll often do almost all the time now is I'll like here I have a refinement here and the way that this refinement happened is I wasn't actually sure what this capture something about a capture policy and how it worked was. So I literally just told I highlighted the section and I do this thing called more details. I don't know if anyone if you guys have clicked on it or not. What what this allows me to do?

35:20 >> Teach me s >> like this allows me to do no >> I'm not locked in because like I live in the t of the terminal and I like I like go back to the massive response and like quote things and I quote things and I quote things and then I like what are annotations like show me your ways. >> Yeah. So I used to do that too and I think for most hacks I can get away with the same thing. But the problem is when I really have to read because again performance engineering nancond level performance I make one mistake everything is and I just wasted my last three days. So I need to I need to micromanage every bite that I'm passing through the system.

35:55 >> Yep. >> When I do that one assumption in the model that's wrong screws me completely. >> So I need to make sure every time I see an assumption like I might be I might be talking about three different concepts over here. Right now I'm talking about like capture precedence like how can a remote how can a remote control plane affect telemetry of your system without you having to redeploy. >> I have promotion policies. How do you automatically take a aggregation event and promote it to an individual span when like error handling happens or when a user specifically denotes that they want it from a control plane of some kind. Then I have await semantics. I want to know how much of my time in a function is spent on IO time versus computation time. Again, very useful metrics. And how do I do this across threads and a bunch of other things? So, I need to be able to maintain context for many ideas at the same time because if I do one idea at a time, it's just one, it's too slow. I can't do that. I have parallel threads running, but then I can't join them together. So, then I run into other problems. So, I kind of need to do a thing where I talk about all concepts all at once. But then I can't go into detail one of these chats.

36:58 And what I used to do with side chats, but the problem with side chats is that they're very expensive. I have to from a UI perspective. So I have to do this, I have to ask inside chat and then I might lose context. It might do other things. This allows me to go into a very quick view that specifically prohibits the agent from actually editing the codebase. It can only read. >> It cannot edit. It's a chat only concept. It has access to my entire chat log. I discuss with it and every time I discuss to it, I literally just go ahead and just say at the very end I'll show Can I show some mine? Yeah. At the very end of my conversation, I just put a message in here that says, "Hey, take everything we have and give me a standalone message that I can paste into the main window. And then I actually have a standalone message." I read this very carefully. Make sure it's actually truly standalone.

37:45 >> And it's it denotes every nuance detail that we talked about and then bring it back in. And then while I'm doing that, that's for small things. But for other things, I don't know, it's not showing me annotations. That's unfortunate. for other things. I'm constantly annotating the codebase and like literally annotating this part of the codebase and say, "Here's an answer. Here's an answer. Here's an answer." >> Yeah. >> Instead of typing everything out in chat. It's a little less context efficient because you end up duplicating stuff because you're duplicating the annotation plus the response. But I find that it's much more accurate for the model maintaining its like context.

38:20 And the last thing I do is every 10 messages approximately, I basically have it restate the entire program like this all the time and I have it restate every single decision that we made, every single key strct repeatedly. So even if it does a context compression, I never lose the state. I have like a I have a very complex state that's going on without having to do any of the risks over here. >> Nice. >> Yeah. I I I remember like I watched a podcast with you and and text a while ago. You guys were talking about performance engineering and you were saying that you need that one million context window because you just lose all of those details, right? I I guess you found a way around it.

39:04 >> Well, no, I still need a million context window cuz like there's details in this one summary that are not captured. >> Yeah. >> Like there are still details here. This is just to minimize the risk of like a context compression by the harness. >> Yeah. Yeah. >> And not only compression, but you need 1 million is really too big arguably for the current models to fully track full attention on everything, right? And to be able to I guess build I like useful stuff you need the or like build reliably. You need to have the key details be I guess to put it in a human term fresh in the mind of the model.

39:59 Even if the entire thing is in the context window having like there are waitings of like this was more recent or this was closer, right? So if you use the annotation tool, I imagine having closer proximity between the restated comment and your response likely increases the association between those two things. >> Yeah.

40:29 >> As opposed to referencing something from >> Yeah. >> way back. Another manifestation of that is is that something that I find personally very annoying like I I use like endless threads. I'm one of those right when my threads don't end and and and so I often you know I noticed something that happened and I I I just want to give some input that is going to be handled I I want it to be handled later. I don't want to don't contact switch right now. keep on the thing that you're doing but get to this later, you know, like and the the the models often just like oh no, this is the most important thing now to do like and they just like drop everything >> and it's like and so now I need to say okay I need to be ficit like keep on what you're doing this is for later you know keep keep a work list whatever like be organized like don't don't be like a >> you know what you know like >> you know what a good hack for that is >> tell me >> just tell the model tell the model you're doing this in a side task Yeah.

41:30 >> And it will literally and if you use the same know if so if you're using cloud tell it you're doing in a fork. If you're using codeex tell you're doing in a side task or whatever your harness calls it because there are tools that the model knows about and that will just let it like relieve stress. I think Perryman has a really good question which is like why not restate it every time as opposed to do it as opposed to doing it like consciously without with the user input. I I'd love to know y'all's thoughts on this, but like my thoughts is like I need to restate it every now and then because sometimes I'm just giving feedback and I want to iterate on my feedback and I don't want to restate the whole thing every single time because that's a slow iteration loop for human time. So I have to balance human time and correctness time with the model in a good way otherwise it just doesn't work for me at least.

42:16 I have one last thing that I think I'd love to Yeah, in context consumption, it just consumes context. I have one last thing that I think might be interesting to actually chat about while we're doing this. we talked about a performance episode. Let's talk about a performance engineering hack and different ways to do things. and like while we did this and like I think it's cool cuz both the model invented this in one context and we had to come up with different rules for this and Tomas you might be able to grill us on this and be like that's a bad design you can make something better so maybe with your expertise we can learn something really new >> perhaps let's see >> so let me open something up really fast I think I have cursor file I don't know why it just opened the freaking ID man and when I do this. I want to just show really fast what I mean by this is like open in terminal.

43:12 All right. So idea of a telemetry pipeline is you should be able to answer these kinds of questions about your codebase for any piece of code you're writing. Can you quickly understand exactly what part of your code is there? So like if you have for example like an actual pipeline that has like stage one, stage two and these are all doing different parts of your codebase. Can you automatically find every function along the way? So like the idea is we should be able to answer these kinds of questions without having to do any explicit telemetry code. Now the problem with doing this is once you do telemetry you have this concept called a span.

43:44 Spans have unique ids. Unique IDs are kind of expensive to mint because if you want to create a new unique ID it's often a UID and a UID is probably on the order of like tens to hundreds of nanconds. Is that kind of what you would estimate Tomas? >> Well, it depends on what version and if you're like getting a clock reading or not. you know, >> let's say UID7 cuz you know those are nice for databas.

44:15 >> Yeah. >> yeah. >> and even UID4 I think is going to be at least tens of nanconds. You're going to hit like probably not because you're still hitting the random number generator. I think >> I built I built a the Go library for >> a unique ID scheme called Ulid. Have you heard of it? >> Okay. No, tell me. >> Okay. Okay. So, so the idea there was to build a an ID that is lexicographically sortable.

44:44 >> Okay. >> and the use case was basically for logs, right? So every log line would would have a one of those ids and and the ids themselves would establish the same order that the log lines should have, right? and yeah at the time UIDs were indeed not the right solution because they were >> expensive for Yeah. expensive. They force force things that we didn't want to eat. >> Exactly. >> Exactly.

45:15 >> Yeah. and then the trade-off is like if you want every function to do this kind of stuff, it gets even worse because now every function call has to pay that overhead and like that basically makes your code like completely dog slow, right? But we still want to be able to do things like that so that we can say things like when I enter the when I enter the geocode function only 96% of them make it through the the next phase. Most of them hit the fallback phase and of the fallback like 3% of them never make it through. Like being able to answer this kind of question is really powerful about your codebase. So the way that we started thinking about this problem is we have a random atomic that is a U64 that's just a counter. Now counters are great. The problem is counters are not shared across your cores. So if you have a multi-threaded runtime like what we do so if you want parallelism we actually have a U64 and then on on every operating system core what we do is every single thread that is running in our runtime needs to every thread calls a bunch of functions. Sorry I'm going to draw some layers.

46:21 Every thread runs a bunch of functions. Now we don't run OS threads. We run virtual threads. So that means threads can kind of hop around cores very easily in the machine. Every core basically at any point grabs a chunk of the U64 name space. So it basically grabs a two to the^ 12 chunk of ids at a time and anything that runs on this core that needs an ID just grabs this ID that's monotonically increasing. So it's just a add increment operation running on every core. If it runs out, then it goes to shared U64, which is behind an atomic. So therefore, like kind of slow to get, and then gets the next chunk of 2 to 12 bytes, and it just keeps on going forever. So you basically get a really cheap ID for all spans and functions in the form of an increment where you pay for more than an increment than like a synchronization point every like 2 to 12 effective counts. Is this kind of similar to what you were describing earlier?

47:25 from from my Ulid library. >> Yeah. >> Yeah. So like we basically the the the random section that is not the time section, right? Like can can be provided by the user. so that that's that's one one thing that we wanted to have so that we can have like >> impotent rights and so the can can hash something that represents other states that that ID encapsulate and >> but this this technique we're not really using that >> I see yeah I what do you think is this fast enough? Do you think we can do faster?

48:07 I mean what's your bottleneck here? It's like the it's a coordination you have to do every two to 12, right? >> Yeah. >> can you avoid that? >> I don't know. That's I I think >> that's a question I guess like so if that's the bottleneck that's a question to answer, right? >> cool. >> Yeah. But in general, one really cool trick that we found while we're doing this was that now we don't have to go ahead and have most of the threads as they're executing don't pay this. And if we find that the coordination tax is too high, we increase this to power 16 to power 17. We pick the number of our choosing. And this basically makes a cost really really cheap >> without having to go ahead and kind of save all the data up front and having to have like a shared a U a unique what we're able to do now is basically every process is able to give a unique ID in the form of the process ID which is a UU ID plus it's like actual like what we will call like telemetry ID for effectively no performance overhead in the form of like an increment amort veris. So what you end up with is a unique ID that you can attribute to any single call stack so that any point in time when you go look at where'd it go when you end up go looking at where'd it go when you go end up looking at any single function call you're trying to understand where it failed or where it passed you have a unique ID for everything but the performance overhead is basically an atomic ad throughout the system so you end up not paying much of a tax but these are kinds of optimizations that we've been able to make in the runtime using some of these these techniques. Now, to close it out, Tomas, I'd love to hear your take. How do you how should someone go ahead and think about performance engineering? Like what should they do? What should they not do?

49:52 >> That's that's pretty broad. >> well, you started a company around this. So, like what's the thesis there? >> Yeah. Yeah. So, like it depends on what performance means for you, right? Like is performance kind of a core part of your business, right? like for instance are you an infrastructure company like building an S3 competitor then performance is definitely part of of your key value proposition right so if you're close to the metal in that way then you're going to have to put a lot more resources into yielding the most out of everything that you're running versus I know some hyper grows company that is doing voice software but they're not really you know like milking every single ncond out of >> of that software red yeah so with birth loop right the the idea the core idea is that the core is that for a large section of the economy like there is immense pressure to adopt a genetic development right and to for and and to harness that capability in probably an unhealthy amount of time like and everyone's trying to do things faster and figure out how to maintain quality at the same time right even before we had agents I think human attention was already limited and we basically applied it to the things that mattered at any given point in time to the especially in the startup right like you you're like f focusing on crisis A and then crisis and Crazy C and and you live with the rest, right? Like and and so that just leaves a trail of a ton of recoverable performance, right? Leaves a trail of of recovery performance that no nobody really gets through. They don't have the time. They didn't have the attention span. They just focus on top one, top two states.

51:57 >> And yeah, Purple is a machine to model your system. hypothesize everything that can be faster to do less work, right? and and let users select what the harness should work through like so like you take one one hypothesis at a time or many and >> on the other side you have basically >> in some ways it's an opinionated harness to basically do this loop. >> Yes. It's it's bur soup. Yeah.

52:28 Yeah. Yeah, exactly. So, so yeah, that's interesting. >> We model a system. We we have a a whole wide catalog of of of performance patterns things like, you know, N plus1 queries and atomic contention and like just a whole bunch of things that we search for on your hot paths that we build from code and telemetry. so like we connect telemetry and code together. Yeah. >> So that we rank what's actually hots up top >> and and then Yeah. And then you can delegate as much as you want or as little as you want to to the system. If you have your own ideas, you can also submit ideas with our MCP server so that you can test 10 of them through our harness instead of >> one or two which is the attention that you have to give to your coding agents if you drive it yourself locally, right?

53:12 >> Yeah. >> Kai, when you think about doing this performance work, what is the number one thing you think about? Like what's a good takeaway for someone else? cuz like I think a lot of this performance work I had someone else on our team ask like how did you know to make this faster in this specific way or how do you know that this is a specific technique that worked and I didn't really have a good answer. It's just like you kind of just figure it out and like you collect like these bucket of tricks and tips of like these are the kinds of pro like like you talk about atomic contention. Generally, if you want to make a really fast system, don't share data across cores.

53:46 That's a really good rule of thumb and it'll be way faster by default. >> Yeah. >> Don't mutate data. Like how do you think about >> this area where like there's really do less work is really the only hard ish rule. I feel like it's also an area similar to like security where it's like everybody knows the goal and the rest is just how much you understand the system that you're working on. and how it works under the hood. because understanding every level of abstraction down to like it's doing a bunch of NAND operations lets you decompose as much as you need. and I think really just like understanding how that system works is the core to it.

54:43 I don't know >> how do you teach someone that? >> That's a good question. I think we've been working on that for about as long as education has existed. is how do you teach somebody that ask questions never ask questions you know just why why why you know why like how like just >> you can like >> the why will take you down to the transistor go down in CPUs like but you know like >> be a toddler and just keep asking why.

55:17 >> Exactly. Yeah. Yeah. >> I think I think a really good technique here is just like forcing people to discuss with other people. I had this conversation with Paulo and our team last night. He's like, "Hey, how do I like you had years of experience learning this? How do I digest this and make this an actionable takeaway for me long term instead of just being like how do I how do I just like do exactly what you're saying?" And like my I thought about it and I think the only real answer I had was like you need to artificially slow yourself down and stop talking to an agent and talk to a human cuz like you if you talk to an agent you can only learn at the pace of speed that an agent is going to teach you but like I it's not like I don't know when you learned performance engineering or Tomas when you learned it. I didn't tell you my experience like I it's not like I I knew all performance engineering when I first started coding. I just kind of picked it up by talking to some great friends and mentors of mine where I'd be like they'd throw me into a problem. I couldn't make it faster. Then like they'd take it over and they'd make it faster. I was like, "How the heck did you do that? What did I miss?" Then they'd describe it to me and teach me.

56:21 It's like, "Oh, this the trick I missed. This what you this how I measured it. These are the tools that I used." And they'd kind of reapply these learnings back to me. And I' I'd take those learnings. I'll internalize and I'd probably remember one of the 50 they told me. The next time I'd get maybe one and maybe I'd figured out a second one on my own because I kind of did some pattern recognition. I'd show them they'd make it even faster and I'm like, "What did you do?" And I slowly just built this bucket of tools just by war of attrition.

56:46 >> Mhm. >> And surviving it over time. >> I I I think that's like the kind of the the motivation side of things, you know, is it's like you I I get motivated by learning with with other people that I respect, right? like and it's like wow you know like you know but I think it's a problem for the future you know like because can can we have like a like a new generation of engineers that is going to spend way less time with people way less time speaking to people and speaking more to AIS like how how how can they get >> more intrinsically motivated in in some other ways >> that's true We're not gonna remove people, right? But but I I I do think that if you already have the motivation, then AI is a superpower, right? It's an in incredible. You can just learn everything you want much faster. if you just keep asking why, you know, if you just keep if you if you just keep asking why and you keep asking >> and you don't just let it go and let it through and whatever. Like no, wait, like this doesn't make sense. Like I need to ask why. And if you probe and probe and probe and probe like you you get instant answers like right like and so >> it's not just instant answers it's actually something better than instant answers. It's like for you like for example I mostly did my performance engineering work all the way in C++ >> I understand C++ pretty well not and that's still not very well because C++ is you know C++ but like there's some things about it that make sense at least on the architectures that I worked on the hardware systems we worked on and then I'm writing Rust now I don't know some tricks in Rust I know the same trick in C++ I don't know how to do it in Rust I don't even know if there's different tricks that are possible in Rust not in C++ ++ and vice versa because like maybe the syntax allows her. It's only nightly in Rust or some weird thing like that like tail recursion.

58:41 >> There's so much stuff in nightly that >> Yeah, exactly. I just don't have the bandwidth to learn all of this, >> right? And an agent lets me do this much faster. Like when we were doing our GC stuff, >> I don't want to invent all of GC like garbage collection from first principles. I literally told it get clone go get clone.net net get clone v8 >> and just teach me how that works. Like give me code snippets, give me all this stuff and teach me the decision that we made and like that is like invaluable and no human can even satisfy that need. But I think the inverse of that is once I've done that, it's really important for me to stop and go to someone else on my team and teach them the same thing I just learned to both solidify my own learning and also help someone else be curious about a different direction of a codebase that they might not have been on their own.

59:31 Cuz like like you said, Tomas, my favorite parts of engineering ever were just the people and I how I interact with them and like how we solve the problem together. >> Yeah. Yeah. that that feels like a a different time of my career, you know, where that happened way more than than these days. >> Yeah. >> Yeah. >> I I guess it's just like earlier on in your career like if if you're learning more from others, you're like you have less experience, right? Like then you go a lot more through that like and then and as you >> as you go >> Yeah. become an expert, it's just harder.

60:04 >> Yeah. Then then it's kind of like it becomes more lonely. a little bit, you know, in the in the sense that you're you're like setting your more, right? Like >> you're spending so much more of your time on like these latter phases that like the earlier phases of learning is like you kind of know the most of the things you can do. So it's like now I just do the work or get someone else to do the work.

60:28 >> Yeah, >> it's like and like Yeah, I agree. How I have one last interesting question. I think there's two very interesting questions on here that I wanted to cover before we close out. question number one is regarding like just continuing off this how do you think we facilitate engineering culture to help with this kind of world like people are mostly talking to agents that's not a bad thing that's a good thing but there's this other element so how do you incentivize people to talk to humans I would argue that being taught off of agents is there is some value but I would not argue that's a pure win Because I feel like and like especially if you're just learning some of the first time or trying to understand core concepts.

61:22 what is it? what's the phrase? The mother invention. >> The mother of what? >> What is the mother invention? I forget what it is. >> I don't know. >> I don't know. It's like you got to struggle through it. >> okay. You think you're not struggling enough with agents that you you should just have people stop. >> Yeah. Even if you understand all the stuff, I don't know if it's I guess good for us to use agents all the time just forever. just for our like maintaining our ability to do stuff.

61:57 >> That's I don't know. I I I I fear that I'll one day become one of those people from Wall-E. >> I emphasize what I have I have like the question I asked when when I hear this sort of argument is like you know what what were like engineering leaders and CEOs and product directors and VPs doing or or senior architects doing all this you know it's like they they were kind of like they were achieving things right but but through other people they were the OG proxies right because because they were not always close to the details. They could if they they needed to and they would go there and they would solve the problem and they you know put out the fire. but you know they they scale their their expertise and their vision through other people and now we have also agents right to to amplify those people right and so I I I think that we're all becoming a little bit more the senior architects you know and it's it's a hard thing to to transition from having zero experience to go there you know it's like and so yeah that is indeed something that I I don't have an answer for at this point like but but >> the thing I I always get stressed about about that conversation when people say that because I actually agree with you.

63:22 I I 100% agree that that's the role. The role is what a staff senior staff principal level IC does at some of these larger companies. But what I I can tell you like what we talked about at Google, most people will never hit staff. That was like a well understood thing about Google. It's like most people will seniors terminal level. It's even possible that a mid-level engineer is a terminal. >> Why wouldn't they like what are the the different contributing factors for that limitation at that in that prior era?

63:57 >> I think there's I think there's a couple of things. One is I think it's just a different workload. It's like you can't take an IC person and make them a manager. A good IC engineer doesn't make a great manager. >> True. It's not guaranteed. And similarly, a great manager may not be a great IC. They're two different skill sets. And I think when you go down to like senior architect level, there's different types of senior architects that different teams have. Some teams have the technical senior architect.

64:25 the persona you have in mind there would probably be someone like I don't know like >> Jeff Dean. >> Lionus is Jeff Dean. Lionus Travoltas like these people who are like legendary can solve the problem probably write better code than most IC's on the team and also have the technical chops to go do everything >> maybe >> and some teams need that because they're a very technical team then you have like broad scoping teams where the difficulty of the task is not about writing all the code but understanding all the systems and how they coordinate together. So I bet many distributed systems have these kinds of people where it's like their job is actually just being like making sure that no fatal feature gets in that'll prevent the critical feature from eventually getting in. And that's a different kind of role.

65:09 >> That's that's kind of a a gardener, right? Like the guard rails enforcers, right? yeah, >> eventually. Yeah. >> And but that's still like operating at the same level as Jeff Dean just in a different kind of capa. Maybe not him, but like you get the point of like a senior staff kind of engineer or principal level IC >> and then you have director level people who are more oriented towards like a product mission. They have a product goal and they have this army of 50 to 80 people under them. They're trying to make them go do something.

65:34 >> Yeah. >> And all of these skill sets I would say are not distributed across engineers in a >> in a but >> it's not a skill set you can learn, right? You kind of have to grow into that skill. You can learn like you can learn. I think like everyone is being forced into is into speedr runninging this now, you know. >> That's true. No, I agree. Sorry, when I say you can't learn it, I meant the wrong thing. It's like >> you kind of have to like Kai said, >> you can't teach it.

66:00 >> You you have to struggle through it. >> Like you don't become a good director by just by like being like, "Oh, here's what I do." And I just imitate. It's like, no, you're grow from like managing an intern to managing a threeperson team to managing 20 person team and like slowly scale it up. >> He's compressed now. like do it faster time like now like you're managing 25 agents, you know, like >> all the time like puppet master. It's like it's like how do you manage these 25 things at the same time? Like you need to like like >> Yeah, >> it's a good way of putting it is there are things that you can learn but you can't teach.

66:39 >> Yes. >> At least not fully. You can give guidance but you can't you can't I mean even most of this right like >> in school >> there are teachers who teach you things but it's not just like here are facts memorize them it's like you have to there's a lot of work that has to go in yourself and if >> I guess with the agents they can't they can't teach you it really either you have to go through that >> it's actually a really interesting model is like the the skill set that's most useful left is like the last like last mile learning like how do you learn the things you have to at any given moment with no prior knowledge and the agent is just a tool to the very end.

67:23 >> Exactly. how how self you can be right like the motivation needs to come from somewhere else if it's not going to come from people making you excited about something right like >> it's this is a new world for introverts you don't have to go talk to people to succeed >> you can one last question then we'll close it out which is a lot of anecdotes and both Tomas you and Kai both said is this is kind of like security it's a very like performance engineering is kind of like security.

67:54 I think what's interesting about security is it seems like agents are doing a lot more security vulnerabilities detection than like humans could because it was a man-hour task more so than like I mean this creative engineering I'm not going to I don't want to underell any of that stuff. It's very very hard. Security engineering is truly hard but it's the intersection of two complex domains computer security and theory and cryptography with understanding the actual system you're operating in. And usually people live in one side of the domain. The best security engineers are like networking security engineers, browser security engineers because they have to learn a domain so richly to intersect it. So agents were ripe for that domain.

68:29 How about performance engine? Is this a thing that like agents just do better cuz they can they can like understand it better? Is this a thing that still has like some known human craft? >> I think that it's while there's a lot that's similar to security. I think security is something where you can have you can utilize the ability to paralyze and have lots of breadth to find security vulnerabilities because like I guess with security it's like the defender has to always win and the attacker only has to win once but whereas with performance it's that the defender always have like or the like you always have to win as the attacker in this scenario or as attacker is probably not the best phrasing, but you always have to win as the person doing performance because if you hit one spot that's like going to slow down your entire system, then the rest of your performance work was pointless. and so I think it's really a flip side where the breadth doesn't win you as much.

69:41 >> Okay. or the the ability to paralyze loss of agents because like maybe there's a bunch of security vulnerabilities the agents don't find because they're not able to go as deep on specific things but they're able to go reasonably deep on a lot of things and so they're going to find some stuff and some of them are even pretty impressive but with performance you have to win every time. >> Okay. >> what's your take? I think that the the unifying angle for me is that they're they're both manable to be seen as just hypothesis and and that basically you have a security hypothesis and it get gets refuted if you can't actually chain whatever chain of exploits was supposed to to give you the golden egg, right? so you need to prove through actual empiricism right that you get to the target that you claim is there and with performance the same thing right like you make a hypothesis under these conditions you fix the question right and you're either going to refute it or you're going to verify it like experimentally right and and so it's it's kind of like if if you think about it as like performance engineering and security engineering as kind of a a lab, right?

71:05 Like where you come up with with these hypothesis and you need to test them rigorously to come up with an answer. I I like that unifying frame for for for these two domains, you know, because they're both very experimental like you can't like like unlike math, pure math, we can't deduce a security a vulnerability through through a lean prover or or or you can >> Well, some people might disagree, but I agree with you. >> Okay. I I disagree. I agree.

71:34 >> I mean, I agree with you. >> I think it's it's it's experimental, right? Like you need actual empiricism, right? Like to to to to get the proof that you need on the other side, right? Yeah. >> Yeah. I 100% agree. The experimental nature is very fascinating to me. I I had my first moment of like I I felt I had complete job security forever as a performance engineer. And then I think Alpha Alpha Go came out. Then after that there's this thing called like tensor matrix. There's some some one of the alpha models did like matrix multiplication faster and they did something that I thought they wouldn't be able to do because they hooked it up to a harness which is I think it was alpha tensor was the paper where they hooked hooked up the training system to the actual hardware that it did and it found better optimizations for every custom hardware for matrix multiplication. Along the way I was like that's it. this is no longer the industry that I want to be in. And that took away all desire for me to want to be a performance engineer. I was like, >> "This is solved." And if it's not solved today, it will only get more and more solved later cuz >> I can't do it fast enough.

72:42 >> It just I I think your role just just goes one or two levels up, right? Like it's like you're not like being like a super optimizer like combining crazy instructions to to find like >> you know like the machines are going to do this because the machines have like >> infinite patience and and and and infinite attention and ours is way more limited our context window right like like way more limited and and so we we got to we got to be the ones like with the taste and like and and with a with with answering the question does this fit into our system the way we want it right like and and so like and that that really greedy is like I think o over time is going to be more and more like reliably done by the machines like >> yeah if I were to go through this entire chat log and have it be summarized. I bet 90% of what you'd find my annotations are is catching fatal mistakes.

73:35 >> My only job is to read here and be like is there anything fatal that's happening here? Is there a fundamental axiom that is introduced being screwing us over or doing something wrong? cuz if I catch a fatal mistake and I know that there's no fatal mistakes, I'm mostly good and as long as I codify the key and that's all we really need. Like one of the things that we had to do was the machine wasn't doing this by default was some stuff needs to be default hidden in telemetry like string.length. We don't want telemetry for that function by default.

74:03 It makes the function way slower with no benefit. So just like telling the machine that hey even though I said I wanted to telemetize everything what I really meant was this like ignore specific built-in functions that just requires me to like really comb through it and like read very very careful about all the code I'm doing. But this was tons of fun. Thank you both for joining. Hopefully you guys had fun as well. >> Yeah, absolutely.

74:29 >> Yeah. >> Should everyone do one performance engineering exercise this week and try and make the codebase faster? Of course. Yeah. >> All right. You heard it here. Go try that out. Try and see if you can go ahead and make your system faster in one dimension. Run an experiment. Run a hypothesis. Try doing it. I guess Tomas's company does this automatically at Perloop. >> Yeah, >> we do this manually with our codebase. It doesn't really matter how you do it.

74:54 Run this loop for some part of your system. >> Yeah, exactly. >> And I think you'll learn a lot by building the system around this. You'll learn the bottlenecks of your system. You might learn a new design constraint you didn't know about that might make your users happier. I'm going to quickly summarize the episode into an outro, but that was it. That was a really fun chat, guys. >> Yeah. >> Do less work. >> Do less work. That's the one takeaway.

75:17 >> Yeah.