How Listen builds AI Agents that review their own work | Florian Juengermann, CTO

0:00 We've been struggling with this for a bit where we used to have our own hard-coded pipeline. Now we completely rethought it. >> Today, I'm talking to Florian Ungermann, co-founder and CTO of Listen. They're known for their agents that can analyze hundreds of interviews, surveys, and focus group feedback to pull out the signal from the noise. Right now, in the main agent, it's not directly file structure. We think of it more as a table. So, the table is every row is a response, and every column is kind of like a question or a feature that we extract. And then, the agent can basically create new columns.

0:29 >> Florian describes how their agent works to put structure to interview responses and gather implicit signal from media-rich user conversations. >> We have been relying more on contextual prompt engineering. There's one feature we have which is emotional understanding based on the video and the audio, not just the text. >> We also go deep into their approach on quality control for the end report the agent writes and their eval system to ensure accuracy. Basically, we have this sub-agent which is this reviewer agent that just knows what a good report looks like. So, that's what we run it using the asynchronous runner. And then, in the live runner, we use it as an evaluation system. Listen has figured out how to run agents at scale, solving some tricky problems on breaking up tasks so that they can be massively parallelized.

1:10 >> We have this hard-coded workflow. If you call this tool once, we spawn those 500 agents, and then we aggregate in a very specific way, and then it returns it. >> Welcome to Max Agency, the podcast that goes deep into how the best agents are being built by builders like you. At Listen, you guys actually have a bunch of agents that make up your platform. Would you mind talking about the different agents you have and what they do and how the user interacts and sees them? So, I think our platform is pretty broad. There's a lot of use cases, and everything is now agent first. The first step is you actually create your product, you create a study.

1:43 Then we have this agent that is, you know, this interactive creation agent that, you know, we call composer. And it really works with a artifact on the side. It can modify this, but it's really like human and AI interacting on the same documents. There's some very interesting research challenges there. >> artifact on the side? >> So the artifact is your discussion guide, which will kind of be Those are the questions we will ask in the interview.

2:05 And this then actually goes as an input to the next agent where she does the interview. So this is the the AI that has a conversation with our interviewee and goes back and forth. And you'll do this with like thousands of candidates or So yeah, so not not necessarily candidates. It's more users or customers we'll talk to talk to hundreds or thousands in parallel. And yeah, this agent it's a little bit less interesting from like a you know, agents building perspective because it doesn't have as many tools.

2:32 It's more like a regular conversation. But it's also like multimodal. It has like image input and you can do screen sharing and those kind of things. Is it voice as well or I think it's that's voice based as well. Yeah. So so that's that's the second agent and then the third big step is on the analysis side and that's I think where we spend most of our time so far. It's building this what we call research agent.

2:51 And that's really like imagine you have you've done, you know, 500 interviews now and those exist like you have the transcripts of the videos. But now you want to explore the data. You have questions about it. You can ask and this research agent is really powerful. It can create everything from, you know, charts. It can, you know, obviously summarize things. It can even like cut video clips for you. It can even now create PowerPoint slide decks in your own company template. And that's I think where we spend most of our time so far.

3:16 Okay, so let's let's focus on that agent. So how do people interact with this? Do they chat with it? Is there some big background job that runs off after all of these and how many interviews are there? You said 500. Is that a typical amount? Do you see hundreds of thousands? Yeah, so in direction it's both. It is the you know, we do run like one analysis run up front and that can take like 30 minutes. Like we we use the same agent architecture for that as for the live interaction chat. There's some different parameters where you want to optimize for latency versus um, you know, just quality. What are those different parameters? Yeah, I mean sometimes we use different models. So, for some like live calls, we use like faster, smaller models, or we tune thinking parameters. So, you know, in the live run, we have like minimal thinking, whereas in the long run, we have like medium or even more thinking. And then there's some There's some things of like if you have more than 500, you have like thousands of interviews, you know, you can't always look at all of the interviews live because, you know, rate limits and and so on. Even if you spawn a lot of sub-agents, there's some limits to that. Whereas in the asynchronous workflow, you can actually look at most of them. So, then we're doing some things like maybe we just sub-sample some and we show you some results earlier. So, there's some slight differences, but the same architecture, um the same kind of agent tools overall for both of those tasks. And for the asynchronous run, like how do you decide what to kick it off with? Will the user set that ahead of time, or is that a standard template that you guys have that you run? Yeah, we know kind of what works best. So, there's a there's a standard template to it, but it's also something you can customize on your kind of organization level. So, if you have a specific format, if you have specific background information, you can input that before, and then it will take that into account. I mean, I think there's like different outputs.

4:55 Sometimes you want to have like a long written document, sometimes you want to have like multiple documents or multiple sections for maybe you're have something that's multi- you know, it's like a study that's run in multi-country, and you want to have like a comparison between those countries. You can specify that up front, and you can have a tailored report to you. But again, if you don't have what you need, you can also just chat with the with it after. And how do you specify that up front? Is that in natural language, or are there some boxes to check?

5:19 >> It's all natural language. It's all natural language. So, you write basically a paragraph of, "Hey, I want a detailed research report. There should be one for each country." Yeah, exactly. And the AI is really good at understanding. I mean, we used to have some some more rigid things, but I think we've all It's all gone to natural language now. Okay, so what does this agent look like under the hood? What is it doing? And if there's differences between the live and and the more long-running one, maybe let's focus on the long-running one specifically, but it sounds like they're pretty similar.

5:46 Yeah, you know, um overall they're pretty similar. I mean, how does the agent work underneath? Um it's actually built our own harness, which is something we can talk about. And it basically has access to these, you know, the main level is the transcripts of these 500 conversations. And then the goal is to you know, really make that understandable. And it has, you know, a bunch of different tools. We're in this interesting domain where it's not infeasible to look at all of these individual interviews again. At least with like cheap LLM. We're not in like the millions of conversations where that's infeasible. So, we have some tools that can do like a, you know, more recursive summarization or even like classification of like, okay, if you want to know how many interviews mention a specific thing, we can actually look have a small model, you know, GPT mini or uh Haiku or something similar. They should look at all the interviews, classify it, and then you actually get robust quantitative data out of these like very open-end conversations. So, how are these files or how are these interviews presented to the LLM, to the agent? Are they presented as files? Are they presented as variables in some programmatic environment? Yeah, so we've been iterating a little bit on that and we're, you know, constantly thinking about if we should change it to more file structure. Right now in the main agent, it's it's not directly file structure. It's more we think of it more as a table. So, the table is every row is a response it's a it's a response.

7:05 And every column is kind of like a question or a feature that we extract. And then the agent can basically create new columns. So, it can say something like, you know, what is the user sentiment towards the specific topic. And then please could be like an open-ended like summary of this the user sentiment or could be a categorical value like, you know, positive, negative, neutral. And then it will basically add this column into this, you know, table and you know, fill in the values for each one of them. And then you can use things like Python or things like, you know, like to to chart the data basically based on that. Super interesting. And so and and when it fills in the values for each row, it kicks off basically a small sub-agent or a small Are are those small LLM, small sub agent, are those the same things?

7:45 Yeah, you could call it a sub agent, but it's really like a very constrained agent. It doesn't have to have, you know, room to decide what to do. It just does like a close classification call. So, that's one of the tools it have is basically like this you can think of it more as like a map produce call. I mean, there's something that that we have hardcoded as one of the as one of the tools. Yeah. So, you can call it a sub agent or you can call it just LLM. So, it's got this table.

8:07 It's got a row for each of the transcripts each of the interviews. It it creates different columns. You mentioned like Python or plotting things. Like does it also have access to code? Can it write code? So, by So, by default we give it some Let's say you call this, you know, this classification tool like with the sentiment, right? And by default this tool actually returns a statistic on, you know, out of the 500 interviews you did, 325 were positive, you know, 100 were, you know, it returns some default values that you can then use and give an give the answer. Most of the time that's actually enough. Or you can use that. You know, we have some, you know, sophisticated way of creating charts based on columns. So, you can say, "Please create a column chart based on this column or a chart based on this column segmented by this column." So, we have some like logic to create charts and visualizations out of the box. But then obviously there's like an like a long tail of unlimited things people would want to do. And that's where the Python really comes into play. So, you can also write custom Python code. And it's a little bit, you know, it takes more duration. It it will not look as nice because it's not in our UI, but it can do more sophisticated, you know, statistical analysis, create very custom chart types that maybe we don't support in our platform. And then it can put either compute values or even create specific charts that we then return and show to the user. Second matplotlib or something like that. What percent of queries do you find need this more open-ended long tail kind of like just raw Python? When we see it using a lot of Python, it's sometimes worth to give it give it a specialized tool to do that task. So, I think the number has been going up and down. The long-running default reports, it actually doesn't use Python that often. Maybe in like 20% of the times. So, you would think it's maybe not as important, but actually it is super important even if it's just 20% of the time because we have this dynamic engine as well, which is I have a specific question and the given tools don't support that. So, then I fall back to Python and actually run their own analysis and and that's very powerful.

10:02 So, even if it's not in the majority of cases, it's very powerful to have it so you can drill down if you want to. >> So, if you're writing this code, does this code run in a sandbox? How are How are you guys managing that? Yeah, so this code runs in a sandbox. We're using um E2B for for execution executing that code. There's some challenges on kind of making sure we get all the data because it's not tons of data, but it's still like a decent amount of data to, you know, spin up the sandbox, load the data fast enough, especially in the live uh live view. So, we do some like pre-warming and like setting up the sandbox ahead of time and those kind of things. So, you have the sandbox separate from the agent. So, the agent's kind of like running, doing its own stuff. It's got this table-like thing, which is uh yeah, it actually is that a real table under the hood? Like is it >> It's purely like a representation. So, I think the agent we we tell the agent that's a table, but you know, in in our database it's actually laid out differently. Uh but you can kind of synthesize it as a table. And then if it actually runs the Python, then it gets it gets basically gets this Pandas data frame, which is basically this table. But it it's never stored as like a table like a Since we fire something, it's just stored in our Postgres in a different format. Yeah, we've we've been experimenting with some stuff like virtual file systems and it seems like this is a virtual table basically.

11:14 >> And so, you've got this agent running, it's got this table-like thing, and then it calls this this tool, and then that spins up a sandbox. So, it's not like the agent's always running in the sandbox. It's got like a sandbox as a tool. And this approach, exactly. So, So, most of the time it can actually just run on our backend and we don't need the sandbox because it's more hardcoded regular tools and then only for for the sake back off initially go to python. We also do have some some newer agents or sub agents that are using the sandbox more natively and definitely thinking about where is the future going? Definitely seems like the future is going to more, you know, agents that have it, you know, are built based on code and run continuously. So it's definitely always something we're experimenting with. What are some of those newer sub agents? Yeah, so one thing and we actually just posted a blog post about this is the powerpoint generation. So we've been working on the, you know, again if you guys take a step back, for our customers often times they the end result is is often like a powerpoint slide deck they want to present. And powerpoint is like weird format where it's not you could just create like an HTML like react page that looks like slides and then you can create a PDF out of that and that's great, but you can't really edit it. Like our customers can't edit it. They want to have something in powerpoint. So we've been struggling with this for a bit where we used to have our own hard coded pipeline where we create templates and then we basically have like text replace in this powerpoint slide decks and image replace and and you know, find the right templates and and so on. So we used to build that pipeline. Now we completely rethought it as we as we have seen the agents being able to use tool calling and especially code code generation to create powerpoint. So basically what we have now is this cloud code agent SDK that runs and writes python code and can then modify the powerpoint file and kind of do a lot of iterations in that. And the way we implemented it so far is it's a sub agent to our main agent. So the main agent decides, okay, I want to create a slide deck. It should have this content.

13:09 It gives all the data and then the sub agent kind of iterates in the code on the code level with the specific skill of how to how to create a slide and then returns the powerpoint in the end. And that's this agent that actually runs in the in another sandbox. It sounds like definitely the sub agent, maybe the main agent, they can they can create like artifacts in addition to giving like the final response. How how how does that work? Yeah, I mean artifact in in some ways it's just, you know, JSON output of a of a specific thing. So, if it's like the basic artifact we have is a chart, which is again in our virtual table, we basically say create a chart. Like it's just a JSON tool call uh create a chart with you know, this column and this column and then we render the chart to the output. There's some other artifacts you can also create like memos or like you know, like basically like a written output that combines charts, combines other things. Can even create things like, you know, cut video clips from those interviews together. You know, based on the transcript it selects these are the some interesting quotes. You have some like retrieval pipeline for that. And then it it wraps that in a specific tool call and then we create this reel and it can reference that in the output. And then we render the reel.

14:12 You mentioned tools and sub-agents already. Do you also use skills? Is that a concept that that's made its way into these agents yet? >> As you can tell like the really getting into like the longer tail of use cases where we've been struggling with, you know, the models getting smarter and the prompts are getting longer and that seems to kind of hold the balance, but if you want to go into the more detailed rare instances where we know this how it's supposed to supposed to work, but the agent maybe doesn't want to we don't want the agent to reinvent it every time, then we're using skill.

14:42 Think in this specific instance we haven't used skills that much yet, but we are been relying more on contextual prompt engineering, I would call it, where like if you want to use a specific tool or I guess there're two things that we have the sub-agents which we're rely pretty heavily on. And then the contextual things of like if it's a study that has the specific data structure, then we, you know, include a specific, you know, how do you compare concepts in the prompt and those kind of things. And then we rely on sub-agents. So, for example, we don't have the instructions how to create the PowerPoint in the main agent, but we call the sub-agent and that has basically the, you know, the skill loaded like preloaded by default with all the all the context. We have some other agents that where we do skills a little more based on you know what we do and I think the the real opportunity is can you reduce the context? Can you make it more dynamic?

15:29 But it also comes with some challenges. >> For the contextual prompt engineering, just to make sure I understand. So that's basically like you'll look at the study that was run and based on those properties, you'll you'll just insert different things into the prompt. And so the system prompt isn't really the same system prompt for every every study or every agent. It kind of varies a little bit depending on the study. >> Yeah, exactly. I mean, I think the main thing is just if you can cut out like a big chunk or like there's there's one one feature we have which is emotional understanding based on the video and the audio and not just the text. But then sometimes some of our studies don't have video and audio, right? And then we can remove all of these instructions from the prompt and then just have it less be less confusing. Again, there's there's no magic trick because in in the worst case, you have all of the cases in there and you don't save anything, so it doesn't really help you in in that case.

16:16 But oftentimes we see that you know, it can help a little bit. How many different tools and how many different sub agents do you guys have? For this research agent, it's probably it's more than you think. It's probably like 15 tools or something like that. And are most of these tools like running something over the table, like a classification or or no? There's probably one or two that that run everything across the table. There's maybe one or two different retrieval modes. There's then maybe one for creating PowerPoints. There's one for creating highlight reels. There's one for outputting specific chart in specific way. There's one for creating like an other artifact like a different artifact. So I think most of them are either output related or like processing related. Like either compute new data or create some artifact that you can then use in the um in artifact. And one other thing we also have is this which is pretty interesting is this feedback tool. Hm. Which especially for the long-running task, you can't really do it in the live one, but in the long-running task, it can then self-request feedback. And so it's basically you have this sub agent which is this reviewer agent that just has a clear context, doesn't have all the complicated instructions on, you know, and then not necessarily sees all the history, but just has just knows what a good report looks like. And then has a little set of criteria, like, okay, it should not have any claims that are not backed up by citations or by data. And then it kind of goes through the report and gives feedback. And that's actually a loop that, you know, runs you know, you can run it quite often and it will actually catch a lot of things and will make things much better. So, that's what we run it using the asynchronous um runner.

17:46 And then in the live runner, we use it as an evaluation system. So, basically, I mean, it we have to make some slight adjustments, but basically, you want to know how many issues does this evaluation agent find in the report and can you, you know, at least the first step is just at least know how bad it is or what the common issues are. And then if you make a change to the prompt, to the model, uh or maybe some change you think is not even related, you'll see if it actually increases things either on our benchmark set or even in production of like if there's a spike in a specific uh problem.

18:16 >> When will you run this eval over the live things? Is it like offline evals that you run before production? Are you also running it over production data? And like if so, is that happening at the same time as it's chatting or like at the end of a day you do a big cron job and score a bunch of things? I mean, I think that's something where we're still optimizing right now. We we just kick that off after after each run in like an asynchronous runner, so it doesn't like block things, but it just gets eval'd live. I mean, there's probably some cost optimization we can do on batch inference, but hasn't been the the biggest priority right now. But it's just fast to kind of have internal internal metrics. And we might also move to, you know, subsampling, but it's the cost is not the biggest concern for us right now. I think it's really can we make the product better and yeah, that it's not the biggest concern right now.

18:56 What does make the product better mean to you? Is it just increasing accuracy on the existing things? Is it spreading out to more challenges and more domains? >> that's something we we also thought about a bit recently because in some ways, if If have the signal all are all the things that are broken. Obviously, you want to fix them and and make progress on those. And that's going to be improving the reliability. That's very important. But sometimes you also want to take a step back and then maybe maybe just optimizing this local minimum. And maybe sometimes you actually want to, you know, change the architecture overall or change the approach overall. And then maybe the criteria is even changed. This hill climbing is very important because otherwise you ship a product that is aspirationally great, but then in practice doesn't work. Again, you can only test it in so many cases yourself. And the customer is the first customer who tries it will use it in a way that you didn't anticipate. So, I think it is very important to try to have those evolve in production. Are there any fun stories of how people were using it in ways that you didn't anticipate when you first launched it or that you're discovering now? There's a bunch of things. People definitely very soon try to like not break it in like a prompt injection way, but like really test the limits of like, you know, can you do like a cluster analysis? Can you do like can you run like a, you know, classification model in the Python side. And then, you know, usually the like the Python Python leverage um image we have is is limited.

20:15 It can't do everything. And then like it tries to like create the code things up from scratch. I'm like, "Okay, sure. Let me write like a classification algorithm like myself in my own Python environment because it doesn't have access to those tools." And then we're like, "Okay, maybe we should give it some some more tools." So, it doesn't have to do those things um manually. It's always interesting and it happens much sooner than you than you expect. When you're like iterating on these insights from production usage, are the most common things you're iterating on like changing the prompt or giving it more tools or updating like the environment that it's running code in? Like where have you seen the distribution of of things go? I think obviously the updating the prompt is the lowest lift. The hard part is, you know, if you change the prompt, it hopefully improves the thing that you want to improve it on. Does it then especially if you're like change it to all caps and repeat it three times, does it then mean something else regresses?

21:03 Right? Because of this less attention to the some and other intersections. I think that's the that's the hard part. How do you gain confidence in that right now? >> Yeah, it's I don't think we have a good solution. Um I think obviously as we deployed and sometimes we roll it out slowly and see, you know, other number of issues, but I mean we have like a example like test set of like maybe 20 different product we can always run it on, we can evaluate, we can see at least on that small sample size is anything and that's what we actually manually look through the outputs and see, okay, still seems good, but it's it's really hard to compare one like five-page output with another five-page output.

21:36 They're different. People would probably argue this one is better, some people would argue this one's better, like I don't know, but at least you will see if something like clearly breaks. How many times, if at all, have you guys like completely re-architected this agent? It sounds like you maybe you're thinking of that now or starting to think of that. >> I've done it a bunch of times. I mean, over the last 2 years, I would say >> And what's what's like been the main impetus? Is the models just getting better or I don't know, you you hear a great talk from someone at LangChain maybe or something like that? But no, like what what what's been the impetus Um I mean, I think the first iteration was just like before kind of agents really worked. Like the the first version was just like a simple rack bot, which I guess probably most companies start out doing and that's still a primitive that we have, but I think both like smaller models getting good enough that you can actually do like look at all the data, classify things uh live, that's been a change and then the big models getting smart enough that it can really orchestrate those things uh has been the biggest change. And I don't think it was like a single thing. I think we've been talking about this a lot and it's been a big investment from on our side to re-architect this and we definitely did a lot of explorations before trying it out and yeah, now I mean, obviously, you know, most people are moving towards like a file system uh based agent, something we we have explored, you know, a couple months ago and there felt like especially in our specific use case, it wasn't like kind of still want to have these specific tools. Is that largely because of like the table that you guys have and the ability to quickly run over all things there.

23:03 >> is is pretty powerful. But, there are arguments against that, especially if like the model companies start to, you know, post um you know, post-train on the specific harness or the specific protocols, it feels like you're kind of fighting against that movement if you have your own harness. I think right now all these like coding agent harnesses are pretty bad at like calling sub-agents programmatically. Like, they can call them one at a time, but what you guys really want, it sounds like, is to basically call it 500 times and have that always happening and this is similar to some of the stuff in like the recursive language model paper, but I don't think any of We're thinking a lot about this. How can we take those ideas and put them into like these coding agent harnesses? And there's some stuff we're thinking about, but I don't think we or anyone has really nailed that.

23:45 Yeah, exactly. I mean, you know, obviously the way we do it is we don't actually the model doesn't actually call it 500 times. It just calls it once and then we have this hard-coded workflow. If you call this tool once, we spawn those 500 agents and then we aggregate it in a very specific way and then it returns it. Can I like call it over a subset of rows? Could it pass in some like filter criteria to filter things out?

24:03 >> Exactly. So, that's that's one of the use cases and maybe also coming back to your question about what have we changed? Like, it's it's prompt changes, but then something like oh, often times it would have to filter. And what it used to do is run it on everything and then write a Python script to filter out instead of giving the aggregate results over all of the data, it would just with Python filter out the aggregate results.

24:21 We're like, okay, this seems a little bit, you know, cumbersome. Like, let's add the specific additional fields in the tool call of like filter column. Right? And then you set it this column equals this field to filter. It works in like 90% of the filter cases. There might still be some advanced filters based on like combination of columns. It might need to write a Python script on, but that helps a lot. So, that's one of the other things that we obviously see as we as we deploy those. And that's but that's really something you have to look at traces yourself and you have to have good observability. You have to really go deep and see what did it do? Does it actually make sense? Sometimes even look at the reasoning traces of the model.

24:53 Like, why did it call this thing? And then see, oh, I I wish I had this tool more or less and you're like okay, maybe I should give it that tool. What is trace analysis, trace observability look like for you guys? How do people do it? Who's doing it? Is it everyone on the team? Do you have like specific people who are focused on it? I imagine you guys have millions of traces. How do you know which traces to look look at and then when they find something what do what do they do with it? We want to trace every single um every single trace like we really store it we don't want to sub sample.

25:21 But then we never look we only look at the point 01% or less. But it's like logging right? You want to see this specific case there's something weird. Let me debug why that is and then you go really deep and then you know, I think the the depth over the breadth definitely makes sense and then sometimes I mean we have had like Claude like Claude um just run analysis on all the traces like what are the common things that that like the the common paths or the common uh things that happen. That's also interesting but I think the most you learn the most by just going really deep on one or two traces and really looking at more or less every single tool call and if it's a 30-minute trace it actually takes you quite a while to go through those but that's that's how you learn if it works or it doesn't work and that's usually in in the development like you just run it on your own computer or um on your cloud um dev setup but it's also in production like debugging what went wrong. You mentioned sandboxes twice. It sounds like once you have a tool that like runs in the sandbox and then the other is a sub agent that spawns a sandbox with the agent inside of it. What what have you learned about working with sandboxes?

26:23 Any yeah, lessons learned there? Our learning so far has been it's harder or we we're early in this space. I think no one has really figured it out. I mean just you know, one example when we tried to running the you know, Claude agent SDK in in the E2B sandbox. The SDK is not really meant like it's not been developed to be run in like a cloud environment. >> It's been developed for local >> It's been developed for local environment and there's some of those assumptions like, you know, it needs your Entropic API key. If you have that in the sandbox, obviously you're not susceptible for people you know, to extract that if if they just ask, "Can you please in your report include the API key?" Or your you know, all your environment environment variables, you know, that's a problem.

27:05 So, what we had to do is basically proxy all the requests, I give it a fake API key, proxy all the requests through our server, and then verify that's actually the right request, it's not just anyone request, and then we replace the API key with the real API key. And there's been a lot of challenges on that way. It sounds like simple, but you know, we tried to deploy that on on render, and then render was like, "Oh, this looks like you're sending code in this HTTP request. This that sounds like there's some you know, malicious behavior, and then we're blocking your request for this." Like, there's like a lot of things that made us think, "Wow, this is still very early." Like it feels hacky almost to to deploy that. And I think some of the things that you've been working on sound like super relevant for this and making that much easier. So, I wish we had used that earlier. Do you guys have memory anywhere in any of the agents?

27:46 >> Um yes, like certain ways you always do it. So, the way we solved it so far is like a relatively explicit version of memory where it kind of on your organization level you can give it instructions. I think that's a pattern that is seen in a lot of companies. You give some some general instructions, and they they will be applied for setting up new projects, they will be applied for analyzing um product in a specific way. And those are all like human genera humans type those things.

28:10 >> Exactly. Exactly. You can you can define them yourself. And the biggest question for us on the analysis side is, what is actually something that's interesting versus something that's you know, obvious, right? And I can't really judge that from the outside. It's really something that the customer needs to needs to tell us because, you know, from the outside everything seems new and interesting, but then when you're inside you're like, "Yeah, that's the thing I I know. I've worked here for 10 years. Like, that's not something new to me." And over time you really build that through all the all the reports we're generating from other projects. You know, how do you how can we use that as an input for creating new studies? And there's a lot of complexity there because maybe someone else maybe you didn't even read this report that we assume we already know because someone another person, another business unit.

28:52 So, it's not super easy to figure that out what is common knowledge, what is not. But, using some things across like you know, formatting preferences or um you know, in the way we set up projects, using some of that kind of previous knowledge that we try to distill. But, yeah, it's it's far from being solved. I think there's a lot more work to be done in that domain. Let's talk about UX for a little bit. And maybe here we can zoom out to the three or actually even like four different agents you have. So, maybe for the the first agent that you mentioned, the kind of onboarding agent, it sounded like there was a doc on like a Word doc on one side and then you would chat with it and it would fill out kind of like the study guide. Could you talk more about that UX and what that looks like? Yeah, exactly. We've been iterating a lot on that. I think the the first version you would you would think of this is you just prompt an LLM to write your discussion guide, your document, right? And then you do that and then you realize it never gets it right 100%. Not not to blame the model. I think the model is great. But, just like you give it one sentence, you expect it to write like a whole page out of it. Like that's probably not going to work. So, there needs to be some interaction. And then basically the second version we built is can we just have the AI can you just chat with it, right? And you can can it make modifications to that, right? And then you have the two basic principles that you have to decide between. It's like either the LLM rewrites your entire document and that works for shorter documents or it works if you're actually making like changes like can you change the tone or something like that. Or do you say like you have some kind of edit functionality that is either string replaces or you have some IDs that you replace only specific IDs, which usually works better, but sometimes can also be confusing if you're making a lot of changes, you stack them, you say like ID number two is now ID number three and then you inserted a new element here and then all the like it's also not perfect, but that's kind of the approach that that that we picked. Can we just make because our documents are pretty long, can we just make targeted changes and so on. That's pretty cool. And then you realize like sometimes chatting with it is not the fastest way to to to modify things. Sometimes you just want to delete that and then telling the AI please delete question number three feels a little bit cumbersome. Or sometimes you just want to reformulate it yourself. So you do you want to have a way of also manually modifying it and you know, OpenAI and Tropicana they have some version of that. I don't think the UI access like supernatural and they've also iterated a lot on those. I've seen like they rolled back some of the changes they did and so on.

31:11 I think we have a pretty good solution now where you're both working on the same edit history. So you can actually undo undo changes and you kind of have the change log. You can compare things and you can manually make changes and then you can also make changes with the with the chat and it kind of knows about the changes you made so it doesn't undo the changes immediately and How how does it know about those changes? Are those inserted How do Yeah, how how does it know about this?

31:32 >> Basically the way it works is every change we make is uh formatted as an edit operation. So you see a log of all the edit operations. So then it at least it knows, okay, you just modified this question. Because the main problem is you can't always have like this is the old appro like this is the old document this is the new document or somehow you need to modify the diff. So we kind of formalize that in like an edit operations way and then the model knows, okay, if you just touch that or if in the history you see that you haven't touched that and you probably don't want to rewrite that. Cuz those are passed like so like if I I I'm chatting with it it gets a document. I go into the document edit something and then I chat with it again. Like that edit is passed in like prior to my message in some way. It's the same like the LM writes out edit edit operations and the human edits also edit operations that kind of fit into the same kind of message history.

32:19 Um yeah. So that's the approach that worked pretty well for us. Cool. So that's the onboarding agent then there's the interviewer agent and that sounds like you've got voice there. It's multimodal. It's more just like a chatbot style thing. Any anything interesting there that you guys have been playing with? The interface like the voice interface is still still not quite solved and we've seen even like open AI like the chat GPT app has been going back and forth where they used to have this like blue bubble that was speaking and now that their voice mode is actually you see the text and it writes out text as well. That's kind of the approach that we have been taking for a while as well because you can actually read text faster than you can you can listen to it and sometimes it can actually be annoying to like you know, I wanted the 2x speed or something.

33:00 Um but then sometimes 2x speed is too fast and actually want to go back. So, I don't think it's fully solved yet and same time for our use case we really don't want to interrupt people in almost all cases. We just want to we listen company right you want to listen to the customers. We don't want to interrupt them even if they may be saying something that's you know, rambling or maybe going slightly off tangent. Often times there is a reason for that and in only very rare cases we actually want to interrupt. So, even if I would take a break for a second to think about something, I actually don't want the AI to jump in. And are you guys using kind of like the the speech-to-text, text-to-speech sandwich or you using the real-time APIs? But now we're we're mainly using the you know, speech-to-text, text-to-speech uh by then just because it's so important to have the smartest models and often times these real-time services and last time we evaluated them at least I think the models are now getting pretty good, but at least when we evaluated them they were like one or two tiers, you know, faster and dumber than they you know, top tier uh Opus and and and so on models that it is so important to ask like it sounds very simple to just have a conversation and then asking the right questions. I mean, guess that's what you're doing today. It is actually pretty hot hot task um and that's why we don't want to compromise on that and and rather compromise a little bit on the um you know, real-time aspect of it. Yeah, I you you've got the fast and dumb interviewer here today.

34:18 Um I feel like most people myself included have largely stayed in kind of just like the text domain of agents. When you think about adding on voice, like how much extra work is that? Is it easy? Is Is hard? It depends. Um I think the hard part is not necessarily the AI. I think it's more like, you know, we're collecting facets It's the interview runs on like millions of people's devices, right? And the more modalities you have, the harder it becomes from a compatibility perspective. And kind of you see all kinds of issues on like somehow the microphone stopped working. Or you know how like everyone used to have like trouble with like Zoom like microphone not getting recognized and those kind of things. So we see all of that across the the globe, right? Globally. So I think those are more of the challenges with the multimodality for us than like on the AI side.

35:06 And the AI models are getting pretty good. I think there's still some challenges on, you know, transcription. You'd think that it's a solved problem. And you know, our models are getting pretty good, but there's still something like you release a new product uh tomorrow, the transcription model doesn't know about, you know, that name. And it probably uses some other name. And you know, the only way we could solve that for now is actually having an LLM that has the context on the interview and maybe even knows some of the terms that might come up. Correct the transcription model basically on the fly just to give this additional like smartness that the transcription model itself can't can't use. So these are some of the challenges. And there's obviously like traditional things of like now we're storing like, you know, thousands of hours of video data and you need to, you know, do that. But those are more like traditional infrastructure problems. And then going on to UX, the final research agent in in the two different modes it has. So when it's running kind of like long in the background, how long does that take? And do you kick one off? How do you kick one off? Does it happen like automatically after all 500 interviews are done? Yeah, that's that's a big It's a big challenge. So again, it runs for like 30 minutes and so on. So the cost is like, you know, significant. It's nothing crazy. And you know, we're usually like a higher-priced uh offering. So it's not the biggest concern, but you don't want to re-kick it like after every time there's a new response or someone updates. And you know, you have 500 responses.

36:25 At the same time, we do want to show you results early, right? You get the first 10 responses. That that's a magical moment and that might happen like 30 minutes after you launch the study or to get your first 10 people to respond, which is very magical. So, we do want to give you something. Um the way we do it right now is we run it at a certain thresholds of of interviews. We do like a full analysis and kind of rethink all the hypotheses and, you know, if the data has changed from 10 to 20 or from 20 to 100 interviews, we actually want to completely rerun this. And that's one part, but then now you have we run it after 100 interviews and now then 101st comes in, 102nd comes in. Like, you don't want to rerun everything. But, at the same time, you have we have like numbers in in our report, right? We have percentages, we have charts and all these kind of things. And those things we constructed in a way that we can actually replace those things. So, they LM never outputs those numbers. It only outputs placeholders. Hm. And then we can run all the classifications, all the Python code again.

37:21 Um but keep the core uh thing the same. Um so, this way we always our numbers are always verifiable and updated. So, you can always click on them. You can see the data that's backed up. And if there's a new response coming in or you remove one response because, you know, maybe you don't like them or it's like low quality, uh the those things will update immediately. Obviously, there's there's like a limit to that. If you say like if in the text it says like, you know, this is definitely the best idea because 80% of people liked it and then more more responses come in and suddenly it's no longer 80%, it's only 20%, then actually the, you know, qualitative takeaways change so that you you do need to run it occasionally. Yeah, that's one of the one of the things we we work with. And, you know, at the end you usually in our use case you usually say like, "Okay, now I'm done." Um or you haven't seen any new responses for 2 days and then we run like a full new analysis. But, I think the real-time component, I really believe in this, you know, delayed gratification. Like, the faster you get your results, like the more uh the better the user experience is.

38:14 So, we we really want to do that. >> Interesting. And then for the And then for the real-time kind of like asking if you if you ask questions, how does that work? Cuz I imagine it's doing a bunch of tool calling under the hood. Do you surface those to the user? Do you hide those? Like how how transparent are you about the agent's work that it's doing? But now we do show like an abstracted version. We don't like our customers don't really care about what happens under the hood too much, but they do want to see something's happening and maybe things are like, "Okay, I'm actually now looking at all the responses again and that's why it might take a while." And we have a loading state and all those kind of things.

38:45 The more tricky thing there is what happens if it now starts writing Python code. Right? Because in in some way, if you ask it a complicated question it says, "You know, the answer is 42." You're like, "Okay, I guess." Uh usually all of our findings are very traceable. Like you can as I said, you can click on the numbers, you can see the breakdown, you can even go down to the individual level of this everyone we classified and why we did it and can really kind of explore the data.

39:11 But the Python, that's no longer the case. And at the same time, you know, no one wants to read Python and in our case our customers will probably also not understand the Python. So, can we make it can we build the confidence that it's the right answer? And obviously there's always been there's always been some assumptions going into that. The instructions are never one completely clear. So, you know, basically we're currently summarizing exactly his assumptions we're taking when we're executing Python in like a little box that you can expand and if you want to look at that, but we don't show you the the raw Python script.

39:45 Which, yeah, is something we're we're thinking about. It's It's kind of in the middle ground. It's not perfect, but it gives you some confidence that what it does is actually right. >> are you how are you summarizing? Is that another LLM call that's like looking at the trace so far and and generating something? >> Exactly. So, after the Python code is run, it will like first we show it as a message just like, you know, running Python. I don't think I don't think we're actually saying that anymore.

40:07 We're saying like running some computations or something. And then after it's written the script and while it's executing the script we actually summarize it to have that text and that's more it's less for like the status what it's doing right now and it's more for you know, where does this come from? Let me go deeper and look at look at where it comes from. When you guys are running like these sub agents or small LLMs over the 500 documents, what if the documents are like really big or really massive? Do you do any chunking there and like further kind of like subsetting the text and and chunking into three things and then running the small LLM over all three things or do you always just treat it as as one big thing? Yeah. So, there's multiple layers. Our interviews are what we call semi-structured. So, we have a rough idea of what people are talking about.

40:50 So, there's there's different sections that, you know, in this section we'll talk about, you know, this concept in this section we talk about this concept or we're talking about different ideas. And so, we basically interviews are annotated that way and we can filter through specific relevant sections. So, that helps. And that filter would be part of the filter that you pass into like the the main agent would decide, you know, for this question we don't need to look at the entire interview, we just need to look at their background information or something like that and then we just cut it to that section.

41:18 It's not perfect, right? Sometimes people might say like, oh by the way, I forgot like they're very different setting. I was like, oh I actually forgot. I changed my mind, whatever. Again, not perfect, but I think that's that's a pretty reasonable assumption. We do use some chunking and some retrieval for if you ask a question like, oh did anyone mention something like this or can you find clips where people talk about this specific topic? Then, of course, in theory we could run this, you know, map reduce function call over all of them, but in practice it's usually hard, but especially if you're going to like the thousands and tens of thousands interviews, you use a retrieval like a semantic search on chunking. We do some hierarchical summarizations as well for this extraction step because sometimes, you know, if you if you imagine you're doing like a a thousand, you know, summaries on individual interviews, that's still a thousand times maybe 200 tokens. That's still a lot of text.

42:06 So, then we're actually using like another layer to summarize that. How has your use of retrieval changed over time? So, it has to change a lot. We're still using retrieval. Again, like the first version of the research agent we built like 2 years ago, it was just a rack. I just say semantic search and then the second thing we added was can we add some robust filters? Can we just if you ask can we just filter for man like we extract metadata we filter based on that.

42:31 That was the first version. The second version we like okay, we saw all the problems and we actually completely moved away from retrieval. We were only doing these like small LLMs looking at everything. We didn't have any retrieval pipeline there at all. But then we realized for some cases again especially if you're just scaling to larger samples, it's still useful and it's sometimes faster and in some cases it's better. But it's definitely less critical than it used to be. But even if it's not that critical it still means we embed everything and you know the cost is not um not prohibitive so we just do that and especially if you're if you're you know, we're working towards a platform where you can search to all of your findings not not scope to a specific product but kind of over time. And then of course the data, you know, becomes much like a much larger corpus and and then I think retrieval will continue to be important.

43:13 I know the coding in the coding world it's you know, people now just use you know, keyword search or grep. Have you implemented that over your transcript? So, we haven't really implemented that. I think the the main thing is in code I guess you have symbols that are like types like it's strongly typed and you can actually search for exactly that string. In like a natural conversation it's much harder. If you want to think if you if you want to retrieve all cases where people are frustrated, that's maybe a semantic search but it's like really hard to I'm sure you can search for a list of adjective adjectives that people might have used but it's really hard. So, in our case we're just sticking with the regular retrieval. On a completely different note, what what does the team that does all this agent engineering look like for you guys? It's engineers right now.

43:57 We're relatively small team but one thing I look for in in hiring engineers is kind of this product sense because I think what there's maybe like have types of engineers that do well in this this world. I think there's one that is like really good at building large scale systems and I have seen that have a good taste and maybe there's something that at Lambs at least right now can't do super well. And then the other side the other side is these, you know, product engineers that really, you know, understand the customer, iterate fast, and you know, try out things. I think it's super hard to whiteboard this is how the agent's going to work. And you know, these are going to be the problems and this is how it's going to work. You you have you need some part of that. You need to have some some idea of where you want to go, but then you need to try out things. How reliable is it? And you need to adapt. And for that, I think it's very important that the engineer itself is the one that that is evaluating, that is talking to the customer, and and and hearing how it's going, looking at the logs and the traces themselves. You mentioned you see is someone else looking at the logs. No, it's actually the person that that built the system that's looking at the logs and trying to understand does the model do what I want it to do. And I don't really believe in people that just prompt like just write the prompt because again, it's a very nuanced nuanced thing of yeah, if you change the prompt, then you also need to change the tools and then all the tools you need to understand you know, that some of part of the infrastructure and and so on. So, I think this end-to-end ownership is how we've been building the product and I think that's where I think it's going to stay the same even as we grow from the team. Do you have any non-engineers contributing to the agent, whether that's a product person or design or some subject matter expert? I don't know. Not directly. So, I think the problem I see is it's relatively easy to change something like the prompt that like, oh yeah, just add a sentence to the prompt, you know, and people definitely want to do that.

45:43 But then from my perspective, I'm like, okay, sure, it's easy to change it, but what about the validation? Who's actually going to take the blame? And if it if it breaks, who's actually going to fix it? Um I don't know. Like the number of PRs we have in our in our organization has exploded. Like it's so easy to write code and everyone is like, oh, can I go get into the code and and write code? But, you know, I don't want the engineers to just be the ones that, you know, review code and approve changes and then later on the ones that fixing it. I think that's that's a little bit of the problem that I see. And we're obviously working closely with, you know, customer-facing um people and we're getting feedback, but then it's typically the engineer that will consolidate with that with all the other requirements and and try to improve that and then get feedback from them again.

46:22 Do you care if people joining the the agent engineering team or the team that works at Do they have to have previous kind of like AI or agent experience or is that something that, hey, if you're like a good software engineer, I think you can pick this up on the fly? I changed my mind a little bit on that. Um maybe like a year ago I was like, we just want very smart people and if they haven't worked with AI, I think that's that's fine, they can learn that. Now I think we're at the time where like it's a bit strange if you've never worked with AI. It's been like 3 and 1/2 years now.

46:49 >> Exactly, right? Or like or even if you haven't worked on your job because like for whatever reason your company, you know, you're not doing it like you should at least be intellectually interested and curious about how does it work behind the scenes and build something on the side or kind of at least know what is Claude doing if I'm asking it to kind of build my PowerPoint, like how does it actually build that? Uh so I think if you don't at least have that level of experience, I think, you know, it's no longer it's no longer a fit. Yeah, I think I'm kind of the same way. I also updated some beliefs in that way. I think that's great. Awesome. Thank you for that.

47:21 Thanks for listening to Max Agency. If you liked this episode, leave a review and subscribe. Send feedback or questions to maxagency@langchain.dev. We want to hear from you.