Tanya Roosta - Information Retrieval in the Age of Agentic AI

0:02 Hi everyone. I'm Tanya Rusta and I work at AMD as the director of AI. Sorry I wasn't able to make it in person, but I'm glad at least we have Zoom. I will be spending the next few minutes discussing evaluation framework for agentic IR. So, let's start with the shift. Basically, what has happened in the last few years is that we have moved from traditional information retrieval, classic rack, where you have the straight line of there is a query from the user, the search engine would get the documents, come up with a rank list, re-rank it, and then present the list to the user.

0:45 to this more of an agentic flow where the agent actually does you know, planning, tries to figure out what the intent of the query is, it does the search, gathers the results, analyzes critiques, and then presents the final result. So, this means that the unit of work has basically used from has moved from lookup to an ongoing conversation with the world. and this agentic IR shows up in the deep research agents as most of you all have seen in the chat GPT and you know, other agents that you're able to put a question out. For example, here is the question is about cholesterol medication and the agent basically tries to understand, you know, what the intent of the question is. Then it goes and uses various tools to gather the websites, you know, different information, pricing information, drug interactions, etc., and then comes up with the answer, tries to ground the answer into you know, retrieved evidence and then finally presents the results to the user. And this basically saves us many, many hours and having to kind of go through all the document blog posts, etc. and trying to figure all of this out on our own.

2:17 As and a lot of times this kind of agentic IR is now taking over potentially 20 turns. So, it has become a multi-hop But here's the uncomfortable gap that still sort of exists. And that is that when we want to kind of evaluate these you know, agentic IR, we still tend to look at the final answer, use scores like blue and rouge to see if the final answer is correct. And given the now the information seeking is really interactive, multi-turn, it has this oral aspect, it has to be evidence-driven.

2:58 simply grading the final answer doesn't work very well. so, what do we do? We basically have to look at the trace and start to you know, look try to assess the correctness, you know, of every hop, you know, was it adequate to answer the user, the cost in terms of number of tokens, latency, etc. And in terms of where the field is heading, I would say there is the reasoning aware retrievals. So, now not only have the keywords from the query, you're looking at the agent's trace reasoning traces. There is the multi-turn hop aware benchmarks that are being developed like M-TRACK. there is the retrieval you know, graph rack so that you know, you can look at the relationships.

3:53 There is self-rag to cut through some of the whole hallucinations and adaptive rag, too, basically, route, you know, queries based on their complexity and effort. And finally, actually looking at the number of tokens that, are used, you know, and optimiz- optimizing around that, as a metric for information retrieval. So, what I want to leave you with is that, you know, our information seeking has changed. The agents have changed how we get information, and that means that we have to also change the way we assess if these agents are doing the right thing and if they're achieving the goal that they have set up.

4:41 And that's, all I have. thanks for having me.