Turn millions of traces into insights with Trace Intelligence (workshop)

0:01 Okay, and we are live. Welcome everyone to another weekly master workshop. My name is Alex. I'll be your host today. I'm joined by Daniel and Eric. Hey Daniel. How's it going? >> Good. Good. How's it going? >> Good, thank you. And how are you Eric? >> I'm doing well, thanks. It's a nice day here in Berlin today. I'm actually so stoked to have you both on a workshop together since we've done a few in the past, haven't we? me and Daniel, me and Eric, and now we get to do a workshop together on trace intelligence because well, this is a feature you two have both been working on since the beginning. Is that right?

0:42 >> Yes. Yeah. Since the beginning of trace intelligence, we've been working together on this bouncing ideas back and forth and figuring out how to do this crazy thing. >> >> I love that introduction already. >> The key point is figuring out how to do this. That's the number one goal. >> Yeah. >> Yes. >> Well, if you're joining us here on Riverside, please take a moment to say hello in the chat. We just like to get a pulse for of who's there and around.

1:08 Eric, you mentioned you're from Berlin. I'm based in London. Maybe if you're in the chat, you can let us know where you're tuning in from as well. Daniel, you're based where? >> I'm in Waterlue, so just like an hour west of Toronto. and we're hosting this. >> It's also a nice day here. It's cooled down a little bit. >> Oh, nice. Cool. Yeah, same here actually because it's been so hot and today the heat broke which is quite nice. Hey, Muel from Copenhagen. Love Denmark. Some amazing technology companies out of Denmark.

1:44 Okay, so we've given it a few minutes for people to roll in and just to say a quick hello. Hey Thomas, you're not far from Daniel's neck of the woods in Toronto. Hey Roberts, good to see you guys. Okay, >> I see Derek says that he has no audio. Does other people are other people able to hear us? >> I do believe so if they're responding to us in the chat about the weather and stuff. >> but I do know that sometimes, by the way, that's hilarious. I was about to try and explain to Derek how to fix the audio with my audio.

2:17 I don't think that would have b I don't think that would have been very helpful. So I'll I'll type in the chat in just a second here. but yeah, let's let's sort kick things off properly here. So welcome everyone to this weekly master workshop. We host two workshops a week on Tuesdays and Thursdays to help you build better agents. Sometimes we're talking about new master features, teaching you how to use them and where you can find value from them in your own products and agents. And over times we're talking with customers seeing how they've built production scale agents so you can learn from their lessons and build even more capable production ready agents. This is one of those very exciting workshops where you get to be among the first to learn about a new master feature, a very helpful feature indeed called trace intelligence. And while sometimes I present these workshops and I teach it at a high level, I think that's great. Today, we're so lucky to be joined by two of the engineers who worked on the feature since the beginning. This is not only going to give you the best possible context, but if you have any questions throughout the stream, please don't hesitate to post them in the chat. And then you get to hear straight from Daniel and Eric how they're thinking about trace intelligence and these latest features. And so, yeah, I'm going to pass it over to Daniel to kick us off with some slides. I'm going to turn my attention to the chat for a second.

3:37 I think Eric, there's some feedback about your keyboard. So, while you're typing, you can mute. Perhaps >> he he he just types like he means it. That's that's a 10x developer. >> That's the intensity and passion with which trace intelligence was coded on a mechanical or or the agent was prompted to build trace intelligence more likely at least in part. >> Some people are saying my screen is blurry. I'm just worried that when I share that might be blurry too. So, let me just give that.

4:05 >> Don't worry. All right. don't worry, Daniel. I'm going to, change a setting here on Riverside that's going to help. And then >> Oh, nice. >> It's okay with you. You can share your screen and we'll kick this thing off. >> Sweet. >> And this looks all right. >> Yeah, looks brilliant. >> Amazing. Okay, so as Alex mentioned, we're going to talk today about trace intelligence. And so trace intelligence kind of if you if you haven't heard of this, we we launched it in beta I think like a couple weeks ago now.

4:46 and essentially it's the next step. It's the next logical step from observability. And so, the whole idea is basically being able to turn your agent, what what your agent already emits into something, that you can understand, basically a map of of what it does. And so your agent already emits traces in the hundreds, millions of like volumewise. And so being able to do something with that data and so kind of mentioned this all already the three of us as you know already so to take a step back every AI application already tells a story and if you want to learn more about kind of the the way we think about observability Eric has done a couple workshops in the past and Those would be good to if you're if you're watching this workshop and you kind of feel like this is maybe not starting from the beginning. It's starting kind of like in the middle like assuming you already know about observability and traces.

6:01 That would be a good kind of like after this to go refresh your your memory about what what observability is and how we think about it. And so we have all these all these pieces of information emitted from every single agent run. And so all that information is fairly dense. We we capture like quite a bit like all the information from every tool call, every step, every processor. And so looking through that information can be pretty taxing. And so, even if you're giving it to an agent, if you're like, let alone you just looking through it visually, looking through passing it off to your agent, your agent can kind of drown in context and can kind of, lose the plot when you're giving it so many traces to look through.

6:55 And so the idea is that trace intelligence reveals issues, behaviors and interactions that never become either explicit feedback from your users or your current observability or learning system doesn't give you any information around that. So it could be either like we can talk about hidden issues, unexpected behavior or real interactions like how people are actually working with your agents and then unexpected behavior like patterns maybe you didn't design for but people are using your agent in a way that might be unexpected, not necessarily bad but just different. So, it's like, okay, maybe you can extract some of that out from your your current agent and build like a sub agent, like a different workflow or even a new product.

7:53 >> And so, I I mentioned this briefly before, but like traces are pretty dense. They they contain all the information from from a single agent run. And if you've some agents are very are quite simple just a single turn asking a question getting a response like a something like a like a doc chatbot like th those are are generally pretty straightforward interactions but then you get longer running agents that can the traces get like quite dense and it's like as you scale that from one to 10 up to a million it becomes almost like near impossible to kind of make some kind of judge judgment based off of a whole bucket of traces.

8:40 >> We've seen in some of the our customers production use cases tra like a single trace can be a megabyte of data and then basically scaling that out even at a 100 traces it's basically impossible for a human to review and understand the different patterns that are going on inside the system and it basically for for humans to review it gets untenable very very very quickly. >> Yeah. And and we found that the hard way cuz we like when we first launched this, some of those really heavy traces broke our system cuz we essentially were trying to pull out all the information from that trace and then it was just so memory heavy on our on our services. So we had to figure out ways to extract it in a way that made sense to to not lose information but still not like poison pill the system.

9:35 And so what we're going to talk about today is essentially you get your your raw traces and then there's some sort of quote unquote magical interpretation layer and then at the end you get a behavioral map about what the agent actually does and trace intelligence. I mentioned it was the next logical step from observability but it kind of exists in this in this loop that a lot of people are building around called the agent learning loop and so trace intelligence is kind of the first part of it and we're already in the process of of building the next piece as well. So what we'll just talk about today is traces, trace signals, themes and the evolution of that.

10:37 >> Yeah. And then I'll jump into it and I believe Yeah. I'll I'll just show the UI and then we'll jump more into kind of how it works. underneath the hood and okay getting more into the >> the nitty-gritty of it. And I assume you're gonna say bump it up. >> And yes, that too. >> All right. So >> Oh, damn. Maybe. Okay, this is cool. We're in master platform right now.

11:07 >> Yeah. >> So this this UI also exists in studio. this is basically just a a copy of it, but this is in in a master platform. >> Okay. So if you're self-hosting master for compliance or something, you can also access trace intelligence. >> Exactly. Exactly. Yeah. And so I'll I'll walk through a few different pieces here. So every when traces come in, we group them together and you get these snapshots. And we'll talk more about what those are later, but essentially you get this you can look for like a a date and time. So you can say like what is it how did my agent look like in the past 30 days? How did it look like in the past seven days?

11:59 Things like that. And you get these snapshots that you can look at. And so this is what the interactions look like on July 29th. And then if I look at August 19th, and so this this agent just kind of we played some traces through July 29th and then we played some traces through on August 19th. And what you get here is that every single trace that comes through, we grade it on these four signals. So goal, outcome, behavior, and sentiment.

12:41 And after they get graded on these, we create a summary of it and then group those together to create themes. And so you can see here, seven traces or a 20% there's are grouped into this urgent order issue assistance. So I could click on that and learn more about it. So these are some examples of okay what made up this this block here and so if I want to learn more about it I can click into it and this is a specific trace. So I can look at like we pull apart observations from the trace kind of inspired by our observational memory. So we use that same idea on on the the trace itself >> and then we have all the different signal summaries here. So this specific trace you could look at how did it fare with behavior goal outcome and sentiment.

13:44 And then if you want to dig even deeper you could look at the full trace as well. >> Okay. >> So this you do you have a question? >> This is really cool. I mean, the thing that's fascinating about agents is that unlike a typical UI where people click the same buttons and text boxes and forms and things, people can interact with an agent in any number of ways just through the freedom of a chat type interface. but then you as someone building an agent, you might not really be sure how people are using it, especially at scale, right? Because to your point, you can't comb through every single trace. So through this, you get to see the most like typical ways people are using the agent. And maybe that's a good thing.

14:23 You're like, "Oh, wow. People are using it in this way I didn't expect. Let me double down on that thing." But it could also be that they're trying something and getting frustrated. And when you see that correlation, you're like, "Okay, maybe I need to improve something here." >> Mhm. Yeah. And and this the same with everything that you you essentially the way you want to debug a problem is you go high level and then you keep on drilling down deeper and deeper the more context you need. So this is kind of like you already had your traces that you can filter on and it's like okay I want to look at traces that had like an error. So it's like okay that that's looking at like a highle view of traces then it's like okay now I want to dig deeper. So this is going a step further and saying now I want to look at different themes or how themes evolve and then I might want to dig into deeper within those. So I I might I'll still look at specific traces, but now I don't have to look at a thousand traces and figure out the the relationship between them or like which traces I actually care about.

15:26 >> Will you talk a little bit about how it works at some point? >> Yeah, we're Eric's going to talk about that after I after I run through a bit more of the the UI. And so we have >> Can you also show like in the seni chart how you can like segment down even deeper? So like you clicked on the urgent order before. If I want to >> can you show how you can filter down inside like the behavior or sentiment after doing that?

15:52 >> I I didn't bump the UI for this. So it's not on it's not on platform. >> All right. but but what what Eric is saying is that you can click into this and then be able to click into any of these. So you can get like this this theme cross referenced with this theme and drill even deeper into okay if I if I have like a an urgent order with a outcome like a bad outcome like I can cross reference those and I was like okay only traces that had an urgent order and a bad outcome and then I just want to have those traces and >> this might a slightly basic question.

16:40 >> I'm I'm not too familiar with this type of chart. You said it's called a Sanki chart. >> Mhm. >> What what's special about it? And what what's the best way to read it? >> the the best way to read it is to understand how data flows from one column to another. And we we played around with different ways to visually show this. and and we do include a couple other ways to visually show it, but this is kind of a good understanding of like what how does everything interact with each other and then how does like different traces flow through the system across different themes.

17:24 The really nice benefit of the seni chart is that like you >> instead of just looking at one signal at a time, you can see how like traces flow that and interact in the and group in the different signals. So you can really bucket different sets of information simultaneously. >> At the moment today, we just have these four core signals, but in the very near future we'll be allowing users to >> define their own signals as well. And that then they can further group down.

17:50 for example, like if you're interested, if your chat agents working on a bunch of different topics, you can make like a topic signal >> and then we could automatically group into different topics that the agent's working on. >> Yeah, exactly. We're currently working on being able to define your own signal. So maybe maybe sentiment doesn't make any sense for for your given agent. It's like, okay, I don't actually want to to score on sentiment. I want to score and like other other different signals. So you can add additional signals whatever makes sense for your your application.

18:26 And then I'll just quickly go through these as well. And so we have this how to like how it flows through at a at a point in time with this one. You could see how it compares over a period of time. And so here it it's not too interesting because I only have two points in time, but you could see the difference in change like from from the first point to the second point, this one gained 34%.

18:59 and then this one here is just looking at each from the theme point of view, how it sustained over the period of of your of your snapshots. Oh, so you can see if it's getting better or worse. >> Exactly. So, so depending on what you necessarily care about, like you want to see how everything interacts at one point in time, you want to compare to two different points in time, or you want to look at from the theme point of view how it persists over time. We have that covered. And I just want to show one last thing very quick. we also have a way so I can ask it.

19:39 can you query trace intelligence for my support agent for the past seven days? >> Oh, how does this work? Is it using the master CLI or something? >> Yeah. So, in the master skill, we have trace intelligence document. So, your agent will know how to query trace intelligence. So if you're not really a UI person and you like to give everything to your agent, we have you covered as well. So you can see here it's making an API call here to the the agent or sorry the the through the CLI.

20:25 >> Oh okay. And this is reading this is using API to communicate with platform in this case. >> Yeah. know similar to all of our other APIs that you can like hit your agents, your your traces, things like that. >> Well, this is so powerful, right? Because once the agent, in this case, master code can access those those records, it can also maybe start to address those things, right? and I think that's my question. If we go back to the UI quickly, suppose I'm using trace intelligence. What what do I do with these insights? How do I use them to make my agent better? What's the what's the first step that people should take?

21:05 >> I think the the lowhanging fruit here my eye would immediately go to okay there's tool call failure alting. So let's look into that and you can get different examples here. So I personally like to go everything through my agent. So once it returns here yeah so it says 32% failed their empty tool call result called generation to halt. So I would I would probably ask my agent can you look more can you grab some examples of what those traces are doing and then dig deeper into those traces and then it can figure out how to apply a fix.

21:50 Is there a way to make this work with data sets? because then you could like capture the behavior and then compare if what you did improved it or made it worse. Exactly. Yeah. So so some of these you you'd probably want to take and pull into so it's like okay I have this example of like a like a tool call failure. I might want to pull that into a data set as like okay now I want to like >> they're just traces right so why not this just this is like your map to >> find the most relevant traces to a scenario be that a failure mode or a use case you care about and then you can create data sets you can run experiments anything you can do with master traces currently you can you can do with this that's really cool >> yeah and that's we're sort of hinting at the next step beyond trace intelligence so after trace intelligence to sort of close the loop on agent learning and more will be covered in the next workshop with you John coming up. That's where you take the sort of the output of trace intelligence, the output of these low hanging fruits or other problems that you want to investigate and then you can make data sets around them in an automated fashion and then try to improve the system.

23:03 >> That's closes the full agent loop. And that's where we're trying to get to to have this sort of like automatic system. >> It's good that your mind goes there because that's kind of I think the logical step for most people. It's like okay now how do I action this and that's kind of the next step that we're building as a part of the pipeline. So you kind of are are hinting that we're okay we're building the right thing like that's that's where your mind is going.

23:29 >> >> let us know by the way Daniel when we should take some questions. There's a few in the chat. >> Oh, yeah. We we can take some questions because I was I was just wrapped up with the UI part and I was going to hand it over to Eric to to get into some more of the the nitty-gritty. >> Oh, nice. Well, some of these Eric has answered in the chat, but just for posterity, let's answer them on the stream, too. Then they'll be in the YouTube recording. so someone asked if these are hotel traces or if they're master native. Maybe Eric, you can just talk for 60 seconds about the difference between OTEL and the way master traces work. And if this is specific to the master version or if it also works with hotel.

24:07 >> So master observability very quickly we do not use hotel natively. We wrote our own tracing stack very specifically because inel you cannot ship spans until they are complete. And in the world where agents are running for a long time 30 seconds or more or very soon with longunning agents agents basically run forever. Not being able to ship data until a system finishes is basically a non-starter. So we built our own tracing system custom up very specifically to improve Asian observability where we can ship information as it occurs. that being said we do export exporting exporting to multiple third parties including hotel.

24:54 So if you want to ship your traces to hotel that's possible. you just have to wait for the the data to close because that's the hotel standard. we also support shipping to many third parties, brain trust, langfuse, arise. There's I think like 11 different custom exporters in the in the system now >> including graphana >> and you can also ship data simultaneously to multiple sources. So if you want to ship data to master platform to use trace intelligence, you can still ship simultaneously to brain brain trust or langfuse or your hotel provider or everything all at once. It's all possible.

25:35 >> That's right. We have a brand new integrations page actually on the master docs where you can spot your stack and then see exactly how it works with master. So I think that'd be a good place to go. there's a question here from Wayne. so they currently use Mastra open source framework and they have an exporter to push all the traces to Langfuse. An exporter is exactly what Eric was describing. You can have one or many. So you can export to master and to Langfuse if you wanted to, for example.

26:03 And they're wondering if they would need to use master platform to benefit from trace intelligence. >> Yes. so you would have to I think I already answered that but yeah basically you should also set up the master platform trace exporter and push simultaneously to your the initial platform plus our our master platform. So we have the data and we can run the intelligence on top of it. >> What's the option to benefit from trace intelligence if you don't want to use platform maybe because you have strong compliance needs and healthcare or something. So we'll eventually have a like self-hosting stack for the entire platform and trace intelligence that does not exist today. If you want to use trace intelligence to today, master platform is the only option unfortunately.

26:54 >> Is there a way to sorry just flicking the chat just the chat jumped as I was reading. and how does someone get started with trace intelligence? Like where should they go right now if they wanted to test this out and use it? I think in the beta they need to reach out to us for to turn it on, but in the very near future there'll be a self self switched a switch to turn on yourself.

27:24 >> I just had a link in the chat so you can check it out. >> In the near future, we're going to be turning it on for all team and enterprise users in the the beta. if you're not on team or enterprise and you want to give it a shot, reach out to us. The best way to get in touch, by the way, is with the contact page. So, I'll just pop a link to that in the chat as well. If this is something you or your team are excited to try, we're happy to hook up anybody on the workshop really. So, get in touch.

27:56 >> And I saw Matt you said question about using click house for storing your traces. Click house is what we use internally for trace storage. So, that's what we would recommend at the moment. Absolutely. >> Amazing. There's a few more questions trickling in. We'll let those build up a little bit and then we'll answer them in a batch in a few minutes here, I guess. Then if Daniel, you'll finish with the demo. Thank you so much for that by the way. It's such an exciting feature and it's one that's so fun to demo because it's so visual. but with that demo out of the way, perhaps Eric, we can pass it to you to share your screen and talk a little bit about how this works on the back end. I have so many questions. I'm quite fascinated to see how you turn those hoes of traces into intelligent insights.

28:38 All right. So, I'll continue talking about sort of the the meat and the magic behind this and the math behind it. so this is just sort of intro. We've talked a lot about this before, but we have all these different traces coming in. Then we do some sort of magic to generate these signals. and these are the four signals we have today. These are like the default ones. So, goal, what is the user trying to accomplish? outcome succeed, behavior, what was actually happening during the execution on the agent, and sentiment, how did the interaction feel? And sentiment is really more geared towards like chat bots where you're having a longer conversation. For an agent where you're just like making one request and getting response and that's the end of the system, sentiment doesn't isn't so meaningful there. Anyway, so how is the trace signal made?

29:33 There's a bunch of different steps that happen. Some of them are code driven. Some of them are model driven. We take traces out of a database. We do some in extraction deterministic extraction pulling out specific information. we like take look at the like the shape of the trace. make like a compressed version of that. We look at the user input. We look at tool calls. try to figure out like what's the intent of the agent based on the instructions and then we create these features. so there's tags deterministic tags. This could be like if we saw that a a the agent wanted to call a tool but the tool call failed or if compared to like the all the averages of the traces that look have a similar shape like this one trace used 50% more tokens than the average or 50% less tokens. So that's like some interesting things to to just point the system at sentiment history. So this is like especially for the sentiment the longunning agents or chat agents. What was the sentiment over time? Did the user have like a a bunch of positive hopeful responses and then like a negative one that if there's like the the sentiment changed from positive to negative then like the previous something probably happened incorrectly in the previous trace or previous run. The last user input. This is just like what does the user say for the last time with like what started the the current run.

31:11 Observational summary. Daniel talked about this a little bit before, but this is sort of like the summary that's showing up in the UI. And this is built off our learnings from observational memory. There's numerous workshops on that if and and webinars about that if you want to dig into that and understand that more. And the intent summary again is like a summary of what the agents tried to do. And then for these four different signals, we have custom prompts that we've worked on for a while to generate these signals. These are generated with LM and depending on the signal we feed it different out inputs.

31:49 So like for the goal it gets I think these these lines are incorrect but like basically the the important thing is depending on the signal different inputs are passed to it and we don't send all the inputs to any one of the the signals we're generating. >> Keep in mind that these these signals that we're generating the the summaries are extremely extremely compact like they'll be only a few words long. And the reason for that is because well Eric will show later but we we have to embed it and it has to to make sense with that. So think that you're you're getting like we talked before like it could be a megabyte of data more than a megabyte of data and you're basically squishing it down into just a few words.

32:37 >> Yeah. Basically four sentences or four bullet points afterwards and one for each goal, outcome, behavior, and sentiment. I'm just a little bit curious that it says at the bottom deterministic where it can be model driven where it must be. what kind of model do you need to pull this off? Like does it have to be an expensive reasoning model or can you get away with something on on the on the cheaper side because obviously we're dealing with a lot of traces here.

33:06 >> Yeah. So for the signal generation we're using a very small model. We were using Mercury Nitro 2 and we've recently changed to use OpenAI GPT4 Nano. >> Okay. >> Cheap now. Yeah. >> Yeah. Okay. Oh, continue. so trace signal generation then we have this like different evidence and then we get this compact signal trace paper. This is sort of like what we're saying like the the output of the tracing is quite minimal. Retrace report export after repeated tool timeouts. So we're like dropping all this other data and just having these like bullet points come out.

33:55 Then how we actually like group these and cluster these and group them into themes. We we take the trace signals, we do an embedding currently using open AI or sorry using MPET I forget the specific version but this gives us for each trace signal it gives us a highdimensional vector 768 dimensions >> and then we use a aligned UMAP projection to convert that highdimensional vector for each signal down into five dimensions.

34:33 And we did a bunch of experimentation here. We found that at like two dimensions there's not enough signal. At 10 dimensions there's still too much noise in the signal to to cluster well. And around five seemed to be the the best for clustering these themes. So we used the aligned to produce the clustering space and they use the HTB scan algorithm to cluster themes. HTB scan is really nice because you don't have to tell it ahead of time how many clusters you expect or if there will be signals that like don't fit into buckets. It can figure that out all on the fly which is really useful for this use case.

35:21 So then like we basically have these little bullet points and we can convert them in through this sort of algorithm we can convert this through and get the the different buckets. so like all the signals that were did a password reset and all the ones that related to billing and report generation stuff and this is a living and semantic model. You saw this in the UI a little bit, but themes in these different groupings can grow in size, shrink. also here like this is just showing like trajectories changing in size but teams can split like if we get start seeing a lot of data for report generation and we can or let's say like this billing maybe if like a lot of stuff starts happening in billing and and it sort of like takes over all the all the questions then the trace intelligence will automatically start splitting this into like billing for system A versus billing for system B and then looking into those more deeply individually. So themes can split, themes can merge, themes can die, themes can be resurrected and you can see that history change over time in the UI and in the system to help understand how your agents are working.

36:47 And then so this how we build the seni chart is basically these are the separate themes and we draw how a trace flows through them. And then when you have more traces flowing through and more and more and more then you can basically draw these bucket of lines and this percentages of how things flow through the system and understand how how traces become this hanky chart.

37:24 So then you can answer the hard questions. Which user goals succeed? Which goals create frustration? Which behaviors lead to successful outcomes? What changed this week? Which workflows are becoming more common? Hi you, John. >> Hello. >> so maybe I'll stop here and see if there's questions to answer. And just to give you a little bit of context here, you John is joining us to host a workshop next week. So we've invited him on to talk a little bit about what's coming up.

37:58 >> So right >> or in two weeks, sorry. and so we'll talk a little bit about that in a moment. I don't I mean there's a question here about how to classify the signals. I think Eric just answered that, but feel free to follow up with a clarifying question if it's helpful. And then JF is asking about if this will be available in studio and yes absolutely the intention is that you can self-host this without master platform in the very near future. So keep an eye out for that and if and I have noticed a few people >> it's available in studio today connecting to your master platform instance if you want to completely self-host it yourself that's coming in the future >> and please do get in touch if that's something that sounds interesting to you. we can certainly give you more details and keep you in the loop in that sense. So m.ai/ AI contact.

38:48 What's next on the deck, Carrick? >> Muted myself. Sorry. The next is just just the Oh my goodness. Sorry. the last slide for you, John, to present. >> Oh, you John. Great timing. Fancy seeing you here. >> Just made it. yeah, so we have an another workshop part of the we'll say part two of agent agent learning coming September 3rd, not next week, in two weeks where we go ahead and we take the information from Chase intelligence such as the themes and being able to use these kinds of these patterns and create issues, data sets and scores for them. So, tune in.

39:41 We'll see you there. Okay, folks. Well, thank you very much for joining us on this workshop about trace intelligence. We hope you enjoyed the demo. It's been really fascinating to peel back the curtain and see how some of this works under the hood. I I personally value that a lot because I want to know how to get the best out of trace intelligence and maybe just feel a bit more of an understanding about how it all works. And then yeah, we are very much looking forward to see you in a couple of weeks where we will continue this story. One thing you should know about Mastra is that we like to release as we iterate and improve upon things.

40:26 And so right now this feature is technically in a private beta. You can find the docs using the link in the chat as well as a link to get in touch if you'd like us to enable this for your platform account. Seriously, just send an email, let us know you're in the workshop and we'll be happy to help. And I have seen a few questions as well about self-hosting and I totally get that as well. this is something that is in in the works and we can keep you up to date with that timeline as well if you get in touch. And this is very much about the intelligence part.

40:54 It's about creating information. And it gives you a map to help you understand your traces. That in its own right is very valuable because now you can separate the signal from the noise, identify failure modes, and double down where your agent is working. So many of us, I think, have enabled tracing because we're told and we've understood it's a good thing to do, and it's better to have those traces than to not have them or realize you need them, and you've lost a bunch of data because it wasn't enabled. that that definitely sounds right and it's a good thing to do, but then we end up in this situation where we have a hoorde of traces and it's not quite clear how to make use of this feature helps you make use of them, but there's always more we can do to help you utilize these insights and that's what agent learning is going to come into the mix. Very excited to share more about that with you in a couple of weeks. But until then, on behalf of myself, you John, Daniel, and Eric, we wanted to say thank you very much for tuning in and we will see you next time.

41:48 Thanks, guys. Take care. >> Bye-bye. Thank you.