Chapter 22.4 - Agentic AI

0:07 >> So final note here, I want to talk about what's called agentic AI. So everyone is talking about this these days. This is it's sort of beyond just standard LLMs. What agentic AI is is typically this idea of having an agent being able to interact with the environment. The agent being itself a large language model, but it really what agentic AI is doing is allowing this model to interact, get feedback, update its model or its course of actions.

0:38 Starts to look a lot like reinforcement learning, which is often embedded inside of these agentic AI models. So reinforcement learning, one of the true success stories of it is really starting to emerge in the agentic AI space. So if you remember, I've had lectures on reinforcement learning earlier, but it is an agent interacting with an environment through a set of actions, making an observation, and then the agent from that observation makes another set of actions, changes the environment, makes an observation, and so forth. And the idea is that if you have a well-prescribed objective of what it's trying to achieve or goal, then this agent can work towards making a set of actions from the observations to achieve that goal. So that's kind of in some sense the idea of the agentic AI agent, which is it's in a time sequence, right? It's it's making an observation, taking an action, getting either a reward from that or making a prediction, taking another action, collecting a reward. And so all of this is done by having a policy, a trained policy, to maximize that reward. So this is always what we do in reinforcement learning is try to in fact optimize towards an optimal policy. What are the actions that should be taken to get the maximum reward? And in this case, the maximum reward is, for instance, if we're going to change this into an AI agent, maximum reward is achieved by accomplishing a goal. That goal could be, for instance, something as easy as solve this math problem where it has to reason through a set of steps to get the reward. So, that's how we would maybe train this and start seeing that we could embed, in fact, logic and reasoning into the AI agent because we are setting up a policy that if it reasons well, then it can actually get a maximum reward.

2:35 So, I want to just point to this survey article here, a survey which is using RL inside of large language models, and you just see this growth here of like exponential I mean, this is amazing. If you just look at the time frames, these are It is moving exceptionally fast how people are using this and to see the uptick in performance. All right, so, science has never moved so fast, science and engineering. What we're What we're on is it's accelerated course partly because these agents are so good that if we can train them, they their performance levels are just, you know, exponentially improving on very fast time scales. So, this is unlike a lot of science where we have a kind of a slow progression of how advancements are made, but now the advancements are coming so fast, you can just see it here on the time axis, right? You know, 2023, 2025, here's where we are in 2026.

3:28 it's amazing, and part of this the reasoning capability of AI agents is coming down to correct embedding of reinforcement learning inside of these models. And by the way, agentic AI oftentimes, is misunderstood or what people call agentic AI is is actually simply a workflow. So, workflows are different than agents. Workflows is you want to accomplish a task that has a series of things it has to do, that's a workflow. In other words, the workflow is using LLM to orchestrate a set of predefined tasks or whatever and code paths to execute a result.

4:11 That's not really an agent. It's a workflow. An agent is an LLM that interacts with a system and based upon the feedback, adjusts behavior to achieve the goal of the tasks, okay? So, the workflow just executes things. The agent actually has to make a set of moves, get results back, and update those moves until it accomplishes a task. So, it's interacting with the system fully and that's a big difference. So, a lot of times when people say agentic AI, they're really talking about a workflow.

4:50 Not really an agentic AI. It's a I call it workflow AI. But, agents themselves have the capability of or are you know, the true definition of it is it interacts with the system. From the feedback, it makes new actions happen. Okay? And there's actually very nice code building blocks and agents out there currently available to you and whatever I put out here, you know, maybe in 6 months this is completely changed, but these are things that are tools available for you to start building agents around, okay?

5:24 So, let's talk about implementing these agents and think about workflows first. So, I'm going to talk about workflows. The workflow has typically an input and output and what I'm going to do is the first way to build a good workflow is by augmenting an existing LLM. In other words, the LLMs themselves, they're very powerful tools, but their performance level is only gets to a certain a certain level. And so, what you can do is modify or augment that LLM by giving it the ability to call tools to itself to in other words, execute from the LLM outwards to get tools or memory or retrieval information to help it make the output better. Okay? So, in other words, the LLM itself now has access to tools to help answer the questions you're asking it. So, this is called an augmented LLM. So, you just give it extra capabilities. So, whenever it's not quite sure about this, it can call a tool, get some more information, produce the output.

6:25 Another workflow is what called prompt chaining. So, you first do an LLM call. It comes out to an evaluation or a gate. If you failed, you exit, but if you pass this, you can do another LLM call to another output to another LLM call to eventually achieve the full output. So, in other words, your output's not achieved by one call to the LLM, it's done by multiple calls to the LLM. And this has a gating agent and ability to evaluate as it's making these these evaluations. So, this is called prompt chain. Okay? That's another workflow.

7:00 Another one is routing. So, you have a router that was able to essentially make multiple LLM calls, look at the feedback, and decide how to weight them. So, for instance, here, there's three LLM calls routed through here and decided that what's coming out of LLM one is the right output to give to you. So, it's kind of an ensemble. Remember, the LLMs are based on probabilistic outputs, right? The most probable event. So, whenever you do these outputs, each LLM call might give you something different, and this is evaluating those to give you the final output.

7:37 You can do an aggregator. An aggregator is a little bit like this routing, but now the aggregator takes all the calls out and determines how to weight them and aggregate them for the output. Instead of just selecting the best one, it actually uses all of them to produce an aggregate output. Or there's an evaluator optimizer. So, you come in, you provide a solution, you have a different LLM to evaluate the solution. So, this LLM evaluates this LLM's output and says, "Well, I don't like that.

8:11 Give me another one." How does it give you another one? You just do the same query again, and it's going to generate a different output. Why? Because of the probabilistic nature of these transformer and attention heads. And then once it gets something accepted, then that's the output. Okay? So, now you have, right, you're training two LLMs to interact. One of them is producing output, the other one's saying I whether it likes it or not. Okay? Now, the agent, on the other hand, is different. Now we get to a true agent. The human does the LLM call.

8:41 The LLM takes an action on the environment, gets some feedback, and it keeps doing this until it likes what it gets, then it stops. Notice, this is actually influencing the environment. Typically, or it's at least evaluating it through the environment. It doesn't have to influence the environment, but it's taking an action in the environment, evaluating that action, taking another action until some goal is achieved before it stops. So, that's the key different. An action on the environment. The workflows don't do that. The workflows just produce the output. They never get feedback from the environment itself.

9:21 Okay? So, that's a key difference, and this is true agentic AI here. And so, an agentic AI framework is actually quite broad in how it would actually be executed. Here's sort of a the scale of what might we might be looking at. So, this interaction of the agent with the LLM, and notice over here when it's interacting with the environment, it's searching files, it's returning paths, it's writing code. It's So many different things are can happen here in this interaction with the environment before it's completed and comes back to you. Right? So, you're basically allowing the agent to interact in very diverse ways with the environment in whatever capability cap capacity it can in order to achieve a better output for you. Okay? Can you write code? Here's some stages of this.

10:13 So, stage one is a essentially a performance evaluation. You're going to score the outputs from from what you might have. This is often does been done done by humans, right? We evaluate the agents to see what do they produce as outputs. You know, I might take these different summaries, right? I could say, you know, summarize a text. And if you just did this 100 times, you get 100 different summaries because of the probabilistic nature, and then people can score these and say, "Well, that's a seven, that's a two, this is a one." And so, the agent has feedback from humans. In fact, that's how we've trained a lot of these refinement tuning and agents is with you know, human feed human in the loop to give the agent feedback. Now, once you have the feedback, now you can use this in the model, and you're essentially using this here to score a a reward function based upon the training you've done here. So, now you say, "Okay, how well I've got this feedback and evaluations. Now, I'm going to use this to train a reward model." Okay?

11:19 Based upon that feedback. Okay? and so, this is a scalar, just determining for instance if you're having it do a prompt completion, just a scalar how well it does. Right? And you kind of trained it over here. You you you human trained it and so now this is thing is actually doing the inference of building a reward model. And finally, you can use this reward model on a reinforcement learning loop. So now that you have a reward structure and you've trained with humans, it now knows how to build a maximal reward by interacting, by taking actions. Right? So now that it has a scoring system available to it and a reward structure, it can actually start to build a model because it has actually what you need in reinforcement learning, which is I have values or reward structure set up. Remember, reinforcement learning works completely on a reward-based hypothesis. And so now once you've done that in step one and two, you know, step one, hand you know, evaluation, step two, now inferring the reward structure, now you can start training this thing and put essentially, ultimately, this agent in the loop to start taking actions, building rewards, and build yourself this the fine-tuning piece you need for the LLM to make this agent work.

12:39 Okay? And here's some more architectures that you might have. And by the way, all of what I'm showing you here is coming directly from Anthropic, which is has agentic AI structures built into their into their code structure or at least you can call on these. And so they kind of have this overall flow that you would use to do fine-tuning of your large language models or to build an agentic AI system where RL, you can see, clearly has this key role to play in terms of reasoning, understanding, and making an agent.

13:12 So, RL comes back to us here in the refinement tuning. And where this is going, you know, right? We're starting to see the rise of agentic AI. So in other words, agents capable of interacting with the environment, looking at the feedback, making decisions, and sequentially making these decisions in sort of what we call a reasoning type of way towards achieving a goal. And so this is much different than an LLM, which is sits there and just once you start unrolling it and just unrolls.

13:46 This is the key piece, especially for science and engineering where we solve problems there, it's starting to have this interaction capability with the environment to pull back the best model possible. So that really starts to structure agentic AI overall, and certainly we're just in the infancy of agentic AI and we'll I'm sure see so much come out of this in the next year or two. And so anyways, keep a lookout for this cuz it's going to change your life.

14:16 >>