transcribe

How to Evaluate Agent Retrieval

Hamel Husain · 50m · transcribed 9d ago
More from Hamel Husain Business
𝕏 Share ▶ YouTube 📥 PDF 🤖 .md

Section Insights

# 0:00

Introduction to Glean and Its Focus on Retrieval

What is Glean and what are they focusing on?

Glean is a leading enterprise search solution that specializes in retrieval. The speakers, Manav and Lauren, introduce themselves and outline their focus on evaluating search quality and the evolution of Glean's capabilities.

  • Glean is a pioneer in enterprise retrieval solutions.
  • The company has evolved from basic search to more advanced assistant functionalities.
  • Understanding the product and data is crucial for effective evaluation.
# 10:03

Evaluating Search and Chat Responses

How does Glean evaluate its search and chat functionalities?

Glean uses an eval orchestrator to run evaluation sets in batches, processing queries through their production system while ensuring that eval queries do not interfere with regular production data.

  • An eval orchestrator manages the evaluation process efficiently.
  • Rate limiting is implemented to prevent system overload during evaluations.
  • Maintaining data integrity is essential to avoid contamination from eval queries.
# 20:06

Optimizing LLM Performance and Latency

What factors are considered in optimizing LLM performance?

Key factors include managing token profiles, measuring latency, and optimizing loop counts to ensure efficient and effective search results.

  • Token management is crucial for maintaining LLM performance.
  • Latency and loop counts are critical metrics for evaluating search efficiency.
  • Search evaluations are more deterministic compared to chat evaluations.
# 30:10

Measuring User Satisfaction in Search and Chat

What metrics does Glean use to measure user satisfaction?

Glean assesses user satisfaction through click metrics in search and interaction metrics in chat, while also analyzing follow-up actions and sentiment to gauge overall user experience.

  • User interactions provide valuable insights into satisfaction but can be sparse.
  • Follow-up actions are critical to understanding user satisfaction.
  • Usage metrics like active users help gauge overall engagement.
# 40:13

Identifying and Addressing Regression Issues

How does Glean identify and resolve regression issues?

Glean monitors logging signals and citations to detect regressions, such as missing logging for data sources, which can impact search results and citations.

  • Monitoring logging signals is essential for maintaining search quality.
  • Regression issues can arise from overlooked data sources.
  • Active monitoring of latency and citations helps in identifying potential problems.

Transcript

0:00 Welcome everybody. I'm really excited about this guest talk from these folks at Glean. If you don't know, like Glean is basically the goat in enterprise when it comes to retrieval. Like they have pretty much dominated the space of one of like the first people to focus on retrieval in the enterprise. And I've always wondered like how do they do evals? So go ahead and kick it off to Manav and Lauren. >> Thank you very much Hamel for giving us the opportunity to present what we do here at Glean. And what we are going to be presenting is work of a lot of great engineers at Glean and years of iteration and experimentation. I'm Manav. I work in search quality. I've been at Glean for 1 and 1/2 years.

0:40 >> Yeah, I'm Lauren. I'm also an engineer on Glean search quality team. I've been here about a year and yeah, super excited to be giving this talk. >> Going on to like how we collect data, eval new work, ranking changes and launch progress. So just to give like context people sort of know what Glean is about. So I'll just go over and give an idea of like so this is like how a search works, you know, you can find all your documents across these different data sources and then in chat, you know, you can ask any questions about a particular topic about your enterprise. You can create docs, you can create slides and sort of this is sort of the interface for the chat.

1:19 And then so like it goes through your corpus, finds the documents, looks at different data sources and then cites the answers over here. So this is just to give an idea of sort of what are we trying to evaluate because I guess it's very important to sort of really understand the product and the data we're working with. But the Glean sort of started off as enterprise search and now we are doing assistant. We are trying to do co-work.

1:42 We are trying to build and help do work for you. So the agenda for today's talk, we'll be talking about how do we build eval sets, how do we run evals, how do we make sure that you know, we can launch a change end to end and also after launching, what other things are we monitoring or looking for? And I one of the questions that we generally ask or even sort of new joinies wonder is like why are we not using more open source retrieval benchmarks? And so I wanted to highlight some of the problems that we have and we can't just import a lot of open source retrieval benchmarks because number one is large indexes. We have customers over billions of documents and these are diverse across tens of 20 data sources and we have almost 275 plus app connectors. So it's really hard to get a good open source retrieval benchmark that has good enough diverse and private data. And then of course like our search is very permissioned.

2:39 You're going to see only whatever you have access to. So making sure that you know our search is correctly permissioned and so a lot of these retrieval benchmarks you have documents sort of access to everyone and you do not know like which document has a access to which person. And also we build a lot of signals from these documents, a lot of visits, a lot of authors like who's authoring which doc, how fresh a doc is. And a lot of these signals get lost in these open source benchmarks. Therefore we need to be creating sort of our own eval sets and benchmarks that can accurately measure and reflect what we do here at Glean.

3:14 >> Yeah, so I can go ahead and talk about how we actually built those eval sets. So we primarily build two types of eval sets. We build some from historical usage data and then some from synthetic data that we produce like based on a document corpus. So I'll talk first about the eval sets that we build from historical usage data. So this is built from real use of Glean search and Glean assistant by our internal users here at Glean. So for info seeking we build a few types of these eval sets. We build search eval sets for evaling document retrieval, conduct selection eval sets for evaling like selecting specific relevant snippets within documents and then like full end-to-end chat eval sets. So first our search eval sets, these map the search query to So you know if you're making a search query for, you know, Monov's Waldo launch document, you want that launch document to be at the top of your search result page. so we're able to mine these from both Glean Search, which is the first thing Monov should in the demo with like a user is typing a search query into the box and then clicking on a document and we assume that the document they click on is the canonical that we should then try to retrieve at the top of the page. and then Glean Assistant, as we've moved like more into the LLM world, it's important to be able to also mine these from how like Assistant is using the search tool. So when Assistant makes a query to the search tool, we take that and then we map that against whatever document it finally cites in like the response to the user. next we also So an important part of Glean Search is not just retrieving like the most important documents for a question, we also try to do context selection within those documents so that like when we return this context to the LLM, we show like specifically the snippets within the document that we think will be relevant to answering the question as opposed to like showing the entire content or no content. And so for this, we also keep track of like what specific snippet within a document was cited by the Assistant in a response and then we use that to build this eval set between the query and the canonical snippet. And then finally we build just like full end-to-end chat eval sets that are based on just like the user's original query.

5:23 Here we don't have a concept of a canonical because we don't really know like whether the historical chat response was actually correct. So instead, you know, this is basically just an input-only eval set of like all of the things people have asked Glean Assistant. Yeah, so I'll go over an example of like given a chat session, how would we mine all of this data? So here we can see, you know, this was like a user query where they asked when did Monov and Kunal launch GPT-4 for Waldo 2? the Assistant does like all these searches where we search, you know, search for the document and then it finds the document and it responds and cites both the document that it used to answer the question as well as the snippet within that document. So to get data from this, we would first for our search eval set, we would take this search query that it issued to clean search and map it to the canonical document that it cited in the response. For the context section eval set, we'd do the same except now we would be mapping to like the actual snippet within that document that it needed. and then finally for the chat eval set, we would just take whatever the input user query was. Yeah. And so now I'll hand it back to Manav to talk about how we actually go about building these eval sets.

6:29 >> So just a little bit about how we sort of have the infra for our eval sets. We log and sort of have all the clean traces. And then once we have sort of these production traces, we have pipelines that are essentially just map produce jobs. And their job is trying to go through the traces, try to filter and tag the traces that we want to run evals on, making sure that you know, we are de-duplicating, we are making sure that removing any sort of experimental or debug traffic. And then once we sort of have a representative sample of the traffic, we then sort of get the actual entries which Lauren had earlier mentioned. And then once we have those entries, we can then build upon our chat eval sets, our search eval sets, and our context eval sets. And these sort of pipelines run periodically, making sure that we have access to fresher sort of eval sets for our engineers to really evaluate their ranking changes or chat changes. And so one of the other things that we really want to make sure is we have the ability for our quality and researchers to make their own info seeking custom eval sets because a lot of times we'll have a change which will be a very specific feature change. For example, in this maybe an engineer is working on specifically adding like a operator filter. And this is basically like when you search, you can do app slack or app app anything that you want to search over or even chat has the ability to sort of trigger these operators. And so let's say you want to build a slice out of these main queries and see like currently how our ranking is doing on these sets. So you can essentially go through our data. You can filter out to operators. This is basically all logged in sequel. And then once you have sort of all of those entries, you can essentially add it to our tool and then get those sessions out of it. And then once you have those sessions, you can then run evals on that specific eval set. And so this helps us allow to really build custom feature eval sets, which is very important when you're working on ranking changes as a lot of times you might not be affecting the entire traffic and you want to scope down to a specific feature.

8:36 And so I think the other type of eval sets that we build are the synthetic eval sets, which Lauren had mentioned earlier. So these eval sets are based on a document corpus and these generally have a canonical answer. And so what we really do is we try to identify some of the important documents based on the clean signals that we have. We have a lot of signals such as when was the document in the corpus, how fresh it is, how how how many people are visiting the document. And then sort of once we have that document, we can use an array of LLMs to generate question answer pairs. And then based on that, we can again create the chat search in context selection eval sets. So this is sort of just like a general process of how we are building this eval set. Once we have the mind top docs, we sort of clean it, we filter it and then we figure out what sort of passages are relevant and then we chunk the passages.

9:29 Like we split each document into different passages. And then based on that, we ask an LLM and also using some of our historical queries so that LLM is writing queries that make sense and is well aligned to the user distribution. How can we create queries that reflect of like what someone would ask about a document or how an LLM would ask about a document? And then once we have those question answer pairs, we do another sort of quality filtering to see if these are acceptable questions and answers. And then once that's done, we sort of create that eval set. So this is sort of just an example. So here we have a snippet of the document and this so this and this is what we decide to our canonical document. And let's say a query that the LLM comes with is like who launched Wall-E 2? So like here we know that the canonical answer is authors Manav and Kunal and LLMs have become a lot better to just sort of create great query pairs and answer pairs from the the document and then this also becomes the canonical snippet for our snippet and context selection eval sets.

10:33 >> So next I can go ahead and talk about how we actually run evals once we've built. So generally like our eval running process we have an eval orchestrator that you know will run an eval set in a batch. So it will load an eval set from storage that we produced with the pipeline that Manav just talked about. Then it will run all of the queries in the eval set through basically just our production search and chat runners and then take those responses and compute metrics from those results. So first I'll talk about a few things we need to do within like our search and chat runners to be able to run these eval queries. So we have some like guardrails when we're doing this.

11:10 So first off if we're running like a large batch of eval queries like you know like 1,000 searches or something through you know our internal production system we want to avoid that from like overloading our system. So we have you know we put all of our eval entries in a queue and then rate limit them as we run them through. We also have various forks within execution basically to keep eval queries from like contaminating like our regular production data. You know we have logging sinks where we try to keep the like regular production queries and we don't want eval queries to contaminate those sinks. We also have some like specific logging and metrics that we might run specifically for eval queries. So logging like information about why documents were ranked the way they were or like debug information. And then we also have some specific things that we try to do for reproducibility. In particular like you know if your eval set was built like a week or two weeks ago it's like a search eval set and maybe the documents have changed in that time. We want to be able to like reproduce the historical behavior as closely as possible.

12:13 So we'll pass like the original timestamp that the query was executed and then we have some like back-end search logic to basically try to mimic the state that the index was in at that time just to keep things like reproducible. So then I can talk about like what metrics we actually use for each of our eval sets. So for our search eval sets, this is just document retrieval. So obviously like the important goal here is that our canonical document or in some cases multiple canonical documents for one query should all be concentrated at the top of the search results. So in this case, you know, in our base arm we see that the canonical document is ranking at the second position and in the test arm it's ranking at the top, which is an improvement. And we have a few different metrics that we compute for this. So we have recall at K. So generally like for instance our search tool typically retrieves like eight to 16 results. And so what we really care about is recall at either eight or 16 to make sure it's actually in that top batch of results.

13:11 And then we also want it to be as close to the top of those results as possible because A, there's just like a more continuous signal to optimize on. And B, typically like we will allocate more like snippets for context selection to things that rank. So we want to make sure that our canonicals are concentrated as close to the top as possible. And then finally we measure what we call CR at K. So basically this just measures like how much the overall ranking changes within the top K positions. Yeah, and then for context selection metrics. So our goal with context selection is given a document we want to retrieve information from that document that is useful to answer the question without like totally overwhelming token budgets. So the two metrics that matter here are snippet recall. So like what percentage of the canonical snippets. So like I talked before about how we get canonical snippets based on what like snippet in the document was originally cited. And we want to make sure that all of those snippets are retrieved while minimizing the size of the result. So like on the left we're retrieving the whole document, which means we get the canonical snippet, but the tool output is like really large, and so that becomes very costly token wise. And then in the second case, we're retrieving fewer snippets, but missing the canonical snippet, so that's clearly not a very good context selection algorithm.

14:28 And we're aiming for this case on the right where we've narrowed down to just the relevant snippets as much as possible. >> Good question. You mentioned CR at K earlier. What what is that? Can you talk about that a little bit? >> Yeah, so it's basically just like an overall measure of like how much the results have like moved from one to the other. Forget the exact like formula we use, but the idea there is that like you have a change that's like shuffling search results like a really large amount, but only offering like small wins. Like that might be kind of risky because like in general, we assume that like our existing ranking algorithm has been like perfected over time, and so we want to make changes that make like small changes to the overall like SERP with like minimal amounts of shuffling, but still get wins.

15:15 >> What does CR stand for? Do you happen to know in this case? >> So it's it's called it's comparison ranking, and it's essentially just like an internal formulation of like a monotonic function that quantifies change for us. And so it gives us as what Lauren said, a of idea of like how many search results really shuffled and changed. >> Yeah, normally we would use like CR to decide like what degree of eval we want to do on something. So like small changes with like a low CR value. Like we have different thresholds for like at what CR value you would want to like run a live experiment on something before launching. So we'll talk a little bit more later on like what live experiments are and when we choose to use them.

15:55 >> And also now since we have an assistant product and a lot more people are now using the assistant product, so search is sort of a tool that the LLM can use. So we also need to figure out how can we evaluate the chat and the chat sessions that the users do. So we have like as we previously mentioned the usage rate and synthetic eval sets, and for each of them we have sort of of different types of LLM judges or different judges that we sort of look at. So for usage based eval sets we generally whenever we have a change we sort of run like a pairwise completeness judge and so what essentially this does is it helps us know if if the new change like how complete was it compared to the old change is it like introducing new information or is it fully complete or is it missing something and this is based on like the rubrics that we have sort of well aligned with golden sets and raw queries and then for the known canonical answers we can run the LLM judge for correctness and here the judge is looking for whether the information really matches what the response had. So really optimizing for whatever the information LLM sites and making sure it matches to what the canonical answer has. So we can break down the response quality here. So on the left here we have an example question. So let's say if a user let say just asks how do citations judge work and so I think so there are sort of three things that we look for. One is like citation quality. What is the LLM really citing? so in our product we give the source and also the snippet and so based on that we can really calculate like the precision and recall here. I guess it's easier to calculate precision here because you can just look at this snippet, you can look at the source and see what line the LLM is referring to and see if it really matches to what the exact source says and then the way we do recall is essentially you know we take what other documents came in the search tool and what was not really cited and what could have been relevant using an LLM judge what could have been relevant to calculate some of the recall metrics and seeing you know if it generally tries to cite all the relevant sources for the specific question that we are trying to ask for. the other paradigm I think that we have started is sort of these agentic judges. So since we have built clean search tool, we have the clean document reader and we have a lot of the other tools that gives access to the clean corpus. We can now just have like a loop of these tools with the LLM and can really help verify responses. And so a lot of our judges have become a lot more agentic judges and it has become a lot cleaner to sort of evaluate responses even and I think it's right now the most high signal in terms of chat eval sets because now they can really just validate the answer.

18:48 And then some of the other one of the other things that we look is tool triggering. So if you see on the left, we sort of show like intermediate steps and sort of what tools the assistant did. And so we can also try to build eval sets making sure that the right tools get triggered when when our clean search tools being triggered sort of filter those sessions out. If there is so essentially like the LLM has the ability to sort of use the clean tool with different sort of parameters, but there could be sometimes where you know your change introduced some confusion for the LLM so that like it's failing every time it's trying to call clean search tool. And so we can look at the failure rates or the tool trigger rates and tool success rates and see how your change sort of affects the error rate for the LLM. And so we also have that metric to see how can we sort of improve our change. And some other things beyond quality at least in chat that we look for number one is tokens specifically because like as Lauren mentioned clean search tool returns 8 to 16 results.

19:53 And let's say if you're working on some change that tries to increase the context selection or try to improve context selection, you want to make sure that the number of input tokens are not increasing by a lot since the more input tokens you increase, you'll have a hit on latency, you are the costs will also increase and generally you would want to keep your LLM context as clean as possible. So it's very important to sort of measure the token profiles looking at the input making sure that we are not introducing too many reasoning tokens for the LLM.

20:26 and then also yeah, looking at latency, how our time to first token and end-to-end latency changes. another thing that we look at and we try to optimize for is loop count because we believe that we have the best enterprise search. And so we want to make sure that every tool that the LLM is doing is intentional and is not failing or getting bad results out of it. And so trying to make sure that these agentic loops and because nowadays like the agents would be running tens, twenties, even hundreds of loops. Trying to make sure that how could have LLM avoided this loop and optimizing, making sure that your change is geared towards that. and then also looking at some of the other rates such as like what what are the different tool invocation rates and the citation rates. So now just to sort of talk about the different tradeoffs and especially like search specific eval sets and chat eval sets. I think our search eval sets or generally search eval sets are more deterministic since there's no element of LLMs and you can easily figure out what the answer is true based on clicks, citations. And I think we have also come at a reasonable state where we can really expose thousands of results to the LLM and run a very high reasoning based LLM model to really go through the entire sort of thousand of results and mine good pairs out of it. And but yeah, of course like search is now a tool that the larger assistant in LLM uses. So it doesn't cover the full stack when we do the search evals, which is why it's important to also see how it interacts with the harness, how the LLM sort of uses the tool. but yeah, we have the problems of like the noisy LLM responses, higher token costs and it's still even with these agentic judges, it's very hard to compare like what a four out of five looks like compared to what a 4.1 out of five looks like.

22:21 >> How do you like to think about balancing between precision and recall for your search? How do you go about making the tradeoff? Is it different for every customer using it? Do they make the tradeoff? Do you make it for them? What do you do? >> So at least for search, I think what our goal has been right now is since now search is more of a becoming more of a tool for assistant, we really want to optimize recall at whatever the search whatever the LLM wants like if it wants 16 results, we want to make sure that our recall at 16 is really good because what we have seen is that the LLM is a lot better in sort of figure like citing even the 10th source whereas the in the click and search world the humans would really just click on the top three documents like literally the first document most of the times. But the LLM sort of actually reads through like the 16 documents that we pass through it. So I think like what we really try to optimize for is recall at 16. So if we can just get it into the 16 that's a good job.

23:22 >> I I love that. so like okay cuz the customer of the search is agents. So you just thought about it differently because now people are you know using chat to get at the search not doing the search most of the time. That's what you're saying. >> Yeah, I guess to build off of that we do also use mean average precision specifically like when training our search models because like you know if you have a document ranked in like the 12th position versus the 10th position like that's going to get the same recall at 16 score. but we do like want it to move up within that 16 as much as possible. so we use like map typically like at training time and it's like kind of a more continuous metric to optimize on. but then like recall at like 16 or eight or whatever our current like num results is is like the top line that we would then track.

24:11 >> Makes sense. >> Yeah. And also I think like maybe Lauren can talk more but also like with regards to context selection, I think we are in a world where the agent can also like have a sandbox and can you know dump the entire search results or documents into it. So it becomes a lot more important in like sort of what snippets we want to tight or what snippets we want to give to the LLM in the context. So, I think there I think we are starting to right now we were a lot more recall optimized, but now there we are starting to be a lot more precise and I know Lauren has been working a lot in making and making sure that the snippets that we choose are a lot more precise and because we are also moving in a world where you don't really want to overload the LLM context and try to reduce the token cost especially when you're dealing with a a lot more frontier models. So, in the snippet selection I think we are starting to move more towards precise and clean snippets and ability for the agent to go back to the sandbox and read the entire document if needed.

25:14 >> And you mentioned that you do have some LLM judges and that you have human labels and or you have ground truth that you're sort of using to calibrate them. How how much effort do you go through to acquire these labels? I understand like some of them are easier than others like you have the synthetic dataset which is you know it's a nice way. Yeah, like what what is involved in like getting some like these labels?

25:37 >> We do have like two types of ground truth. The one is like the synthetic LLM one but of but I think like sort of like with the newer models and every time like a new model releases it's very easy to just saturate on synthetic eval sets it seems like. So, now with regards to like how do we get the like golden ground truth? A lot of it is just either like just like company sourced, crowd sourced making sure like we go through the data ourselves and make sure like the judges label work well. And now we are also like sort of building this like continuous loop optimization systems where we sort of as we are looking through these eval entries if we see like a bad eval entry we sort of notify the LLM that you know this is like a bad eval entry and then it optimizes the rubric again and then it has a regression set also so that like doesn't regress completely. I think like I think just to give more context right now like at least search and assistant engineers they're sort of expected to also like always analyze at least like five to 10 entries make sure that like it's still like whatever the labels are there like it's still aligned to what one would expect and there would be like sometimes where you know as a company or even like we would have people sort of analyze just different evals and just give us some of the ground truth data because I think one of the other thing is like it's harder for us to just hire like some external labelers because there are a lot of like enterprise specific domain things that they would need to know and which is why right now we have just sort of worked on just like company sourcing and >> I love that you make your engineers look at five to 10 traces that's super powerful.

27:22 And like how do you convince them to do that because a lot of places people try to say that but then no one does it you know thing >> Yeah so essentially every time you sort of work on a feature or you work on an improvement we sort of have to make this sort of launch document to make sure that the team is aware about it but essentially what it is is like making sure that all the metrics are correct for people to really understand to make sure the change you're making makes sense and also like that's where you sort of add your analysis of like five to 10 queries. So it's it's not really just any queries generally like we recommend to you know analyze like wins and losses and I think at least we have good ways of measuring like this is where the diffs have happened. So if you know like where the most diffs have happened you can just go there and at least try to see. So in search it's easy like you can since we have like let's just see our metric we can like just see like okay this is where the search ranking has changed the most. So just you know seeing like why the search ranking has changed the most just reasoning about it. For chat at least with agentic judges we can cleanly cleanly tell like when an answer is completely different or at least how different the answer is and now it's just like trying to make sure that whatever positives, negatives the judge says aligning with that label and so like using the other to measure. I think the next thing we'll talk about is just like launching a change end-to-end and sort of how are we making sure that we can validate the impact with the end users and how are we sort of testing our changes and what sort of sort of signals we are looking for. Yeah, so we train we have a lot of custom ranking models in our let's say embedding models or our re-ranking models and so we sort of train those ranking models on the collected data. So we have like per document signals and then we sort of log those document signals at query time and then we have like these clicks and citations data and then once we have sort of that data we can sort of train new architectures or we can also add new signal and backfill and see how that would have looked like with the new model like what would the new score be from the model and sort of like iterate like that for our like re-ranking models.

29:29 Yeah, so I think as we have already covered I think the first step for us as engineers always trying to run offline evals first and making sure that the change is geared towards what you expect. So like pure ranking changes we can just run search for context selections changes we can just run context selection evals but for anything that has a lot more interaction with the harness itself it makes like we have to run like end-to-end chat evals to really get an idea of the full impact. Once we have like offline evals run and all of the things have been approved we sort of go for the AB testing where we sort of split users across different experiment arms and we look at the user interactions between each arm and compare our satisfaction signals and so by satisfaction signals what do we typically look for? So in the click world in the search world you know we used to look for things like where the user has clicked has the user clicked at all. So that would be sort of our way of measuring satisfaction in search.

30:32 In chat it's metrics such as like upvote, downvote, has the user clicked on maybe let's say copy response or as you just shared the response. So a lot of the user interactions, but a lot of these user interactions also like sparse. So this might be like very high signal for few sessions. But sometimes like a lot of times these won't be statistically significant to lean too much into it. So we might also have to look into actually like what the user did after the first response, like what were the follow-ups? Was the sentiment positive or negative? Did the user continue the session or not?

31:07 So generally of course like if the user is still saying the same thing in rephrased it completely differently, that's a bad session that we should be catching. But if the user just starts talking something completely different, that could be like a sign of good good satisfaction, but we would also want to make sure that we look at how the first response was answered and make sure that it makes sense to classify that as positive. So a lot of work right now is just you know, how can we make sure we can mine clean and good signals from the user follow-ups and what the user did in the session.

31:42 And then we also have like some of the other top line metrics such as usage metrics. Is the user logging in? How have we changed clean active users, daily active users, weekly active users? If if it's like a longer LE session, how have we like impacted cost on the user side of things? So those are some like metrics we look in like these AB experiments. >> It's really obvious to me that you all are really good.

32:09 This is probably the best example of a real company doing EVAs the right way or that I've seen. And that means a lot because I've seen like many different companies. So a lot of people always want to know like how do you structure your team with regards to EVAs? Do you have a separate team doing EVAs? Is everybody doing EVAs? Is there like an EVA expert? How do you go about it? How do you get everybody or how does this work at Clean?

32:35 >> So I think right now we are at a stage where everyone is doing EVAs. So we do have an eval team and the eval team is sort of responsible for making sure that the infra works correctly. the running of the queries are correct, you know all the production and the infra stuff. We have a specific dedicated team that works on that and we also have another team that sort of works on how can we make eval tool better for engineers? how can we like work with like search team, assistant team? how can we take feedback from judges and improve judges? How can we maybe like go from pairwise judges to list wise judges? So we have another like sort of team sort of working on that specific set of problems. But within like let's say the search assistant whenever we are working on feature or any improvement also, I think like everyone is sort of expected to look through the data and see like what sort of things they are impacting and then once you have sort of launched the feature, how can you like create a good like eval set for anyone to use in future? how can you create like a golden eval set so that it can be used as like a regression eval set for future chat changes? and then to some extent you know I guess trying to build upon the eval set for that specific feature itself. So like for example, just to give an example like I think we had built a lot of like embedding retrieval synthetic eval sets and then once sort of like we saw like gains on those eval sets and we launched the feature out, how can we like then like have those eval sets to go to production and then make sure that like it's maintained well? So that so that's what I mean like you know sort of like everyone who works on a feature and improvement is sort of also expected to make sure that their feature is well represented and has like a golden set for future changes or future regression sets to catch upon. But yeah, Lauren can add more that I may have missed.

34:32 >> Yeah, yeah, I think that covers most of it. Yeah, I would say you know we have like our our eval team that handles infra and just like make sure everything runs smoothly, but like the onus is generally on the engineer who's launching a change to make sure that they are like building the like whatever eval set or runner is needed to like get metrics for their change. And I think also like we just generally have a pretty metrics driven culture, I would say. And so you know, if you're trying to get something launched, you need to be able to prove that it improves metrics. And so there's sort of an incentive to like figure out the best way to actually measure the effected slice so that you can get that to work well.

35:11 >> That's great. Yeah, I mean it seems like you're doing both. You have like a centralized eval team that's like like infrastructure and you're making everybody think about evals at the same time. Yeah, it's curious. I wonder if this level of focus on evals and this level of rigor is like something to do with the fact that you all started with search. And you know, search tends to be very metrics driven. So maybe that culture just continued.

35:35 >> For sure. I was going to go through like an example of how we would launch a change. So something we did like several months ago was a pretty like large revamp restructuring of like our backend search architecture. So we were training like an entirely new model to replace what we had before. So this was a pretty big change over like several months that required like multiple steps of like training eval and iteration. So the first step obviously is training the new architecture. So for like our training data for search, we generally use pretty much the same data that we then use in evals, of course with like a split so that we don't mix data in between the two. But it's the same like click and citation data mapping like queries to canonical documents. And so using that we can get like a very large data set to experiment on. So then like completely offline like fully detached from like the main like production search pathways, we can do like experimentation with you know, any architectural decisions input features etc.

36:33 And so that's just like you know, classic ML work. And then we arrive at some kind of eval candidate. And so once we have this candidate, we'll move into using our like production evaluation system. so this is once like the new architectural change has been kind of like merged into production behind a flag gate. so that we're able to like make sure that the code works fully in production. So we run like large search evals and look at the same metrics we've been talking about. So top line search metrics, things like latency. Want to make sure there's not like regressions for specific data sources or anything weird there. and then once we have like a finalized launch candidate in search evals we start with search evals because they're cheaper and deterministic. Then we can move on to chat evals to fully understand like end-to-end impact like once it's like in chat. so the hope there is that things like loops, latency, and token count will go down because if you're able to retrieve like the relevant results more quickly, hopefully the chat will be able to produce a response more quickly. And then also hope that quality metrics either, you know, improve or stay consistent. Yeah. And then I guess as Manav was talking about, we then have live experiments typically after eval.

37:49 And so the main advantage of a live experiment is just that like the sample size is much larger because this is just real user sessions. And there's also more like quality feedback than in an offline eval where we might not have like either it's user generated data where, you know, we don't have a clear canonical answer or it's synthetic data which might not be fully representative. So in live experiment we can measure like real-time satisfaction signals from our users to things like upvotes and response copies and so on. as well as, you know, the generally cost and latency metrics.

38:22 >> Yeah, I think it's also important to talk some of the other like real-time metrics that we sort of track and this is to like inform a lot of our changes and in some way sort of helps how we sort of evaluate new features or new development. So what are some regressions outside of launches, right? let's say we launch something, are we making sure the user's satisfied? Let's say if the user's satisfied, did the cost increase or decrease.

38:47 so like I think as we already mentioned we really track all of these metrics to make sure it makes sense and have alerts if they don't and then that helps us also inform whether you know a particular offline eval set really helped us or not and so that's how we sort of build intuitions out of it and I think like that's sort of important to figure out you know when would a particular offline set correlate to like the end user metric. So for example like just to give an idea of an intuition like our click search eval set we know that's a lot biased towards clicks that it already has received because that's what the user was seeing. We wouldn't expect huge changes or huge improvements in click eval set unless like it's a very major change.

39:33 generally like click eval sets have been more like sometimes an indication for us like if we are significantly regressing or something and then we would rely more on like some of the other synthetic sets some of the citation sets on LLM to see like if the change is improving there or not and then once we know that okay like satisfaction has improved by that much because we saw that much gain in the citation eval set we have a better intuition on seeing like okay what percent of like satisfaction can we expect when citation moves by that much.

40:02 so that's that sort of stuff really like helps us like go like the end sort of regressions or like looking at the satisfaction at the end and how can does it correlate to the offline eval sets and then how can we build intuitions out of it. So we can go through like an example regression which was basically due to dropped logging. So we really rely on signals lock signals for our re-ranking and retrieval and so if like a logging is missed for a given data source that it could completely be that we just don't see that data source in search or the assistant just doesn't cite it or doesn't see it at all. and so one small example we had is we had a my we had a regression where we were not locking logging slack documents correctly like the slack children threads correctly and so were not log citations based on that. And so that affected like our sort of that affected our the citations that we had on slack trade data source itself.

41:01 And this is something that wasn't as cleanly caught in offline, but when we were looking at tracking metrics per data source, we saw that okay, wait, why slack not being cited enough? That's then that's when we sort of then looked back and sort of figured out, okay, we have a missing signal logging. And then how can we make sure that you know, our offline eval sets then get improved and making sure that we are also making as we talked about, you know, looking at tool trigger rates, looking at like different app also like how making sure that like app rates, citations within different apps doesn't fall to a certain degree.

41:37 Yeah, we also have like making sure that you know, we catch latency regression where the smallest of changes can cause huge use of latency regression in our experiences. So making sure that we actively monitor search latency, chat latency, looking at different percentiles and have a good grasp of what your change would do to the distribution. And I think we are also in a world where you know, we can't just route everything to the best model possible. So it's really important to route and figure out how to correctly calculate cost. You know, some some models are very good with image generation slides, but that still requires info seeking and stuff. And so that will also involve the use of clean search tools. So therefore, it's very important to see like how a particular like feature trigger has changed over time, whether image generation slides have changed over time and how does the cost profile look there? And then within search also like you know, looking at like index reliability and how the costs change there. Cool. Yeah, so what's next? And like what are we doing? What are we building for eval sets as well as how are we sort of improving our flow?

42:46 So yeah, evals has always been like a cat and mouse game. You build a particular eval, you start doing well at it. A new model drops or maybe saturating eval and now you know you have to keep on iterating through the evals. So at least that's definitely a pattern with synthetic. With sort of the usage based evals, it's more like a feedback loop, but you also want to avoid the case where you know like as we already mentioned we try to avoid. We split the training data and eval data, but you don't want to be really mixing the two or like avoiding like loops that are detrimental for development. So I think like one thing we are doing is of course building larger and cleaner golden sets, building more representative data, feature specific data, working with different teams and making sure we have the right data, whether it's usage, whether it's synthetic, but scaling up for each and every feature and query slice. And then now we have like decision based modeling. So we can at least cleanly go and really create good subsets out of these data. And once you have the good subsets out of the data, an engineer or researcher can quickly run a lot of different subsets that they think they have affected on and then run evals on. I think we are also working more towards reproducibility because Glean has a lot of like write actions, a lot of creating docs, creating slides.

44:08 And so when you're running these chat evals, how can we make sure that we simulate those actions as correctly as possible based on that time, that users, that level of documents that they had. We are also working on how can we cleanly iterate on online signals because you want to make sure that we get a clear engagement data out of it. And then another thing that we are really focusing on is multi turn evals. And I think mainly it's working on this simulation loop and if you do a change, how can we make sure that the entire 10 turn eval is correctly simulated. At what point, you know, we'll have deviances and we want to sort of figure out okay like why did it deviate. And we are working the eval team is working really on how can we simulate multi-turn evals correctly and as accurately as possible.

45:00 So I think this is what we have for the slides. I think the best way to sort of learn what evals is trying to construct an eval set on your own and because that really gives you perspective on like what you're trying to eval, how the infra looks like, what the metrics you want to track and then once you sort of build changes, how how does it look on the eval set? I think like at Lean generally in search team we have all built eval sets for our features or improvements either custom or actually made changes to the infra side ourselves sometimes, but that's been the best way sort of helping us really ground ourselves with the data or the query or the feature that we're working on.

45:41 >> No, this is an excellent presentation. Like I said, you know, I've been teaching evals for the last three years and we've had many guest talks over the years. People always ask me, what is the best example of someone doing evals? >> Mhm. >> This this talk is going to become the best public example now of someone doing evals. I mean it's it's very good. I'm really glad that you shared this. I think a lot of people are going to learn from this. You know, it covers a lot of ground, everything from like, you know, synthetic data, offline evals, the search component, how that goes downstream. You also talked about product metrics. I mean it really covers like the whole gamut and it's very comprehensive. How do you go about hiring people because, you know, it's evident to me like, okay, you're really your team is really good at this eval and they have a lot of rigor. Is there certain skill sets that you look for? Do you hire try to hire people with a certain background that would be, you know, that you find that are better at this? Do you think about this in a certain way or is it just yeah, can you talk about that?

46:43 >> Yeah. So it's a great question for my manager, but I think like at least when search what I've seen is people who are good in like thinking quantitatively really helps definitely. And so at least like some of our interview rounds is like just, you know, talking about like probability statistics or just going a little deeper on how you think about these things. And I think it's just like basically skills to sort of understand and quickly realize okay like how could you understand the metrics much quicker. It sort of helps with that. So that's a definitely one sort of skill I think that our team looks for, just like quantitative analysis. So at least like I think some of our interview questions have been just like like you know something like as simple as like talking more about like binomial distribution or chain rules or something like that. So just strengthening the basics of probability statistics and basic calculus for sure. I think one other thing is just like with search once skill, which is a little harder to see during interviews, but will do you wonders in search is having the ability to build intuitions. And it's important to really like have a hunch, try things out, eval and then like either succeed, fail and quickly work on your intuition.

48:02 And I know like the best of engineers at Glean within search are ones that have intuitions about a particular stack and how they can quickly iterate through and either try their intuitions out. So I know like for example like just like a very basic example is it could be like you know you're working with search queries and you are starting to realize that oh wait like we have like like let's say it's like someone is typing the launch dock and like why are we really attaching synonyms to the and then being like okay like this is like a useless word or like how like why are we attaching bigram synonyms to this. And so like how can I quickly like drop these synonyms, quickly like run a ranking change. So like always sort of questioning like you know like testing the query out, why is the ranking in certain way. Like having those sort of curiosity moments and those moments where you are like having some intuition about the stack. I think that will sort of help you a lot in making like good changes within search. I think at three just being passionate about the new. I think like since like LLMs are there, I think you can definitely like scale a lot of a lot of the synthetic stuff quickly. You can like really get it to go through like trace and spans quickly.

49:15 Like it can read a lot more at the end of the day, but you should be the one driving that analysis and having that intuition to get the change out. >> Sounds great. Okay, we'll end it on that note. Thank you so much, Manav and Lauren. This is an excellent presentation. I really mean it. It's like one of my favorite maybe my favorite presentation on evals as a guest in this course. So thank you so much.

49:37 >> Real quick. If you're building an AI product and you're stuck on how to improve, we put together this free resource on evals to help you. It draws from more than 1,000 hours of working with companies on building AI products. When you open the guide, it'll look like this. And what you do is you select the stage that you're at. Maybe you're new to evals, you don't even know what evals are. Maybe you do have evals, but you don't really know what to test. Or maybe you don't trust your eval scores. Or maybe you think your product is just too hard to eval or it's too time consuming or expensive. Whatever the case may be, we have a detailed guide customized for what stage you're at. Go to bitly evals-guide and we'll send it to you.

Summary

Manav and Lauren from Glean presented an in-depth overview of their evaluation (eval) processes for enterprise search and assistant technologies. They discussed how Glean builds and assesses eval sets, emphasizing the importance of custom benchmarks due to the unique nature of their data and user permissions. The talk highlighted their systematic approach to collecting data, running evaluations, and monitoring changes post-launch to ensure high-quality search and assistant functionalities.

- Glean specializes in enterprise retrieval and has developed its own eval sets due to challenges with open-source benchmarks.
- Eval sets are created from historical usage data and synthetic data, focusing on document retrieval and context selection.
- The team employs a rigorous process for running evaluations, including rate limiting and maintaining data integrity during tests.
- Metrics used include recall at K, context selection accuracy, and comparison ranking (CR) to assess changes in search results.
- The evaluation process is iterative, with engineers expected to analyze their changes and contribute to the creation of golden eval sets.
- Post-launch, Glean tracks user satisfaction signals and operational metrics to ensure the effectiveness of changes.
- The team emphasizes a culture of quantitative analysis and intuition-building in hiring, aiming for engineers who can effectively iterate on search improvements.

Questions Answered

What is Glean and what are they focusing on?

Glean is a leading enterprise search solution that specializes in retrieval. The speakers, Manav and Lauren, introduce themselves and outline their focus on evaluating search quality and the evolution of Glean's capabilities.

How does Glean evaluate its search and chat functionalities?

Glean uses an eval orchestrator to run evaluation sets in batches, processing queries through their production system while ensuring that eval queries do not interfere with regular production data.

What factors are considered in optimizing LLM performance?

Key factors include managing token profiles, measuring latency, and optimizing loop counts to ensure efficient and effective search results.

What metrics does Glean use to measure user satisfaction?

Glean assesses user satisfaction through click metrics in search and interaction metrics in chat, while also analyzing follow-up actions and sentiment to gauge overall user experience.

How does Glean identify and resolve regression issues?

Glean monitors logging signals and citations to detect regressions, such as missing logging for data sources, which can impact search results and citations.

© transcribe · For agents Built with care and craft by Gokul Rajaram