Shiva Kasiviswanthan - Toward Adaptive Agent Frameworks

0:02 Good afternoon, everyone. My name is Shivakashi Nathan. I'm a principal applied scientist at AWS. Me and my team are working on developing AI agents for monitoring and observability for large-scale cloud operations. And today in this talk I'm going to share some broad research directions that we've been exploring to make agents more adaptive. So, I think like I'll maybe start like this. Today's agents are already pretty capable, right? So, we all know that how how good they are. They're good at planning, tool execution, retrieving information, like iterating over the solutions to come to solve complex tasks, right?

0:38 But as these agents get deployed in more mission-critical applications, we in AWS believe there are some critical research questions that need to be solved because they can be widely deployed. And here are some of them that we've been exploring in our team. The first question that we've been exploring is this question of like how should agent allocate computational resources, right? Imagine that you have a fixed computational budget and you want to agent to best utilize the computational budget to solve the problem at hand.

1:05 The second question is along making agents reason better, right? So, agents needs to be able to explore multiple reasoning paths to to make a decisions in a more efficient fashion. And the third research question that we've been exploring is the question of how do you evolve agents over time, right? So, the agents are not static pieces, but that's continually adapt over time to make and learn how to solve the task over time better. So, in this talk I'm going to just focus on the first two tasks, but I'll like just briefly end up with how we are solving the looking at the third problem at the end.

1:39 So, let me start with the direction one. Like so, this is the notion of like budget-aware adaptive execution, right? So, so today if you look at most most of the agents, they have a fixed execution policy that you define using a prompt or like some other local heuristic, right? I guess what our goal is to to make these agents learn these execution policies. So, so the agents can learn over time these execution policies so that the and and then they decide what how the the these policies are then executed and like then you also have notion of like a computational budget.

2:08 Like you want the agents to work best under some computational budget. Right? So, abstractly you can think of this as a problem where you have some context. The agent has some context and it needs needs to decide the best next action to do. the action could be like a real information, use a tool, verify the results, update memory, whatever. Like there is some action space. And then and the the important constraint is that there is notion of budget, right? I mean so there is a budget that agent has and it has to respect that budget while while executing these these tools, right?

2:39 So, again like I mean if you think of this like a little bit more like I think this can be like phrased as like a in in in terms of a formulation it can be phrased as a constrained Markov decision process. And we've been using this as a framework to to learn these policies, right? So, again the goal is to maximize some kind of expected utility under some notion of subject to some computational budget, right? Again, the budget could come from latency, cost, token, or whatever you want to think of like some notion of budget.

3:07 Right? And does it work? And yeah, the answer is yeah. I mean like we've been have some success in implementing this. so, what we've been shown that like this plot like on the on the plot on the right hand side shows these adaptive agents learning better execution policy than using a scalar reward model which is commonly used in in practice. Right? In other on the on the the other plot on the on my left side is is showing that it's also more sample efficient, right? So, you're you're reaching the same competence using fewer iterations. So, the the the amount of samples you need to get to this some level of competence is significantly lower if you use these kind of adaptive execution policies. If you learn these adaptive execution policies.

3:48 Okay, the second direction that I would just briefly want to talk about is notion of parallel reasoning, right? So, so today we have these agents or at least these models which are following a single reasoning path, right? And we have been exploring ideas that allow models to explore multiple reasoning paths in parallel. So, these these parallel paths are kind of learned during training and not during inference time. And then they are coordinated to select one path, right? So, the answer is one. So, there is some coordinated reasoning that happens at the end which takes these multiple reasoning paths and produces one answer.

4:20 And again, we have been implementing these ideas and testing it on benchmarks and like both over mathematical reasoning and coding benchmarks we have been getting consistently good results. Like and these are some results on like some math and coding tasks and you can see the results for pass one to pass K. We have been getting better than like very strong baselines. And especially the results are impressive on K equal to one which is like like the first one shot one when we don't want to get the one result out.

4:49 Okay, so like maybe like I'm almost out of time. So, I just want to finish by going back to the research questions that I was interested to say. So, we have been like I just mentioned about this budget-aware adaptive execution where the goal is to learn an execution policy that's adaptive and best uses the constraints it has. And then the question of at the second question about parallel reasoning where you're trying to generate agents to have multiple reasoning so that multiple reasoning paths so that they explore multiple paths to solving the same problem.

5:15 The third question that we have been also been focusing is those notion of continual model or agent adaptation. Again, the goal is to come up with principal post-training schemes that can adapt to new task. The the main challenge there is to make sure that you when you learn the new task you don't forget what you learned previously, right? And we believe that these kind of like these solving these kind of three research directions is will help us to generate the next generation frontier agents.

5:40 Thank you.