CMU AI Agents 2026: 2. Tool Use for Language Model Agents

0:00 Okay. So, hi everyone. welcome to class number two. Today, today I'll be talking about tool use for language model agents. And, this is, you know, the the most fundamental thing about agents that, you know, makes them different from a language model. so the basic idea here is that we're going to be a tool is basically an interface through which a language model can invoke an external computer program. And this is a language model cannot do this on its own because it's fundamentally a token predicting machine. But this enables you to do you know many many different things. So you know you can search, you can call a calculator, you can call a browser, you can call Python, you can call arbitrary APIs and so the language model kind of as you'll see the method for tool calling is standardized but what you do downstream with the tool call can be very different.

1:09 So there's basically two benefits of tools and all of this is from one of the readings that I I posted on the website if you want to go back and read more about it. this is a survey that I we wrote together Daniel and I and Zoro Wong, one of our our students. and the first thing is extending the ability of language models. So this is giving language models the ability to do something that they fundamentally cannot do. And so this could be like accessing information or actions that are outside the language model's you know action space. The second thing is facilitating the language model. And so what I mean by this is this is something that fundamentally the language model can do but it's just a lot easier if you don't if you don't require the language model to do it.

2:02 So can anybody come up with an example or well I guess we already have an example in here. So like one thing the the language model fundamentally cannot do would be get the time four months ago or something like that. So the most recent information it saw is from four months ago and then its parameters are frozen. So there's no way it could know that without calling you know an external function. On the other hand, something in the realm of facilitating is language models can do math actually quite well nowadays, but they take a lot longer than calling a calculator to do math because they have to do reasoning like you ask them to multiply two seven-digit numbers and it will take them a very long time to do it where a calculator can do that really quickly. So those are the two kind of basic categories.

2:52 So to go a little bit more deeply into those you can do things like look up current information and the benefit of this is this gives you you know fresh evidence to ground your answers do exact computation using like calculators or python this gives you the ability to execute more reliably or and quickly than using the language model itself. other things is access private states. So giving the model access to like information about the user so they can respond based on the user state or make a change to your external environment. So this could be like browsing or or calling an API.

3:33 So to have some examples of these if we think about chat GPT chat GPT used to like literally just be text in and text out when it first came out. it, you know, the the only thing that it could do was give you textual responses to your queries. Now, it's kind of an everything app. It allows you to do so many different things. so like what what are some of the examples of things that you know are your top use cases of of using chat GPD now?

4:10 >> Yeah. >> Writing code. Okay. So what what tool would you need for something like that? like cursor or codeex I think they they can invoke like >> okay but that's not chat GPT right that's cursor or codeex yeah I mean chat GPT like in the chat GPT interface >> they can use blender so they they can render blender okay things so that that's an example of something that requires executing code but also like displaying it in the appropriate format any any other ones that people do yeah Just the web page generate images.

4:49 >> Generate images. Yeah. So that that's calling an image generation tool. Any others? >> Yeah. Yeah. So recommend products like a blender or something like this. And for this what what sort of tool do you think it would need? >> Some sort of like web searches like Google or Amazon to find. >> Yep. So you might need like arbitrary web search or you might need an API that directly links into Google or Amazon or something like that. Yeah. Any other like weirder weirder ones? Things you were surprised chat GPT was able to do?

5:31 >> Yeah. >> You can render latex quite nicely. >> Okay. messed up files when I fix you the PDF >> and compile everything which is very handy sometimes. >> Yep. Ren, it can render latex. Well, I actually don't know if this is a tool or if this is just a front-end interface thing, but it might be just that it's rendered in your web browser. >> Oh, it creates a PDF. So, okay. Is a tool. Okay. Very cool. Yeah.

5:57 >> Interact >> interact with like Slack or something like that. So, that's another specialized API. So all kinds of different things. I I kind of categorized these into a list of things that I think are the most like typical tools and most of the other tools can kind of be categorized into each of these. So the first one is textual responses. So this is kind of the boring one where it just gives you an answer. there's two ways you can implement this within an agentic like loop. By the way, chat GPT is is an agent now. if you're like if there was any question about it, chat GPD is not just a chat app. It's an agent because it makes iterative tool calls to answer your queries. So that's kind of the definition of an agent. but textual responses, there's two ways you can implement these. One way you can implement these is you can have a finish tool that says, "Okay, I'm done interacting with the user, so I'm just going to finish and and output some text." The other way you can do it is if the agent decides to not call a tool, you can just finish the agentic loop. So that's kind of you know the the simple u you know most obvious one.

7:11 Another extremely common one. so all all of these are tools and so like theoretically all of them can be thought of as an API but just to break them down a little bit more. another one is searching the web and this is you know pulling in information from all of these different settings. how many people have heard of retrieval augmented generation? Probably everybody everybody who took the NLP class here you know has heard of it but a lot of people. How many people have heard like rag is dead?

7:46 A fairly large number of people. So there there's been an argument in like language model circles that retrieval augmented generation was so like two years ago and like nobody cares about it anymore. My argument here is a retrieval augmented generation where you retrieve context and then generate answers based on it has become so normal that people don't even realize that they're using it anymore. And so it's it's so alive that people think it's dead. but basically you know searching the web and retrieving information and then generating answers based on it is now essentially the norm within these these applications. Now so that's another tool that's provided very very broadly and like any chat interface.

8:29 Another one is code execution and so this is throwing in you know a set of code to do calculations or or something like this. very useful for computation all the other stuff we talked about. Another common one is having some sort of image generation and this this is you know obviously another very popular use case in chat GPT. this one's a little bit different because like all of the other ones here that I was talking about basically you know mostly interface with text. here this is interfacing with this is interfacing with images. One interesting thing is you'll notice that this is not the same thing is here.

9:16 So this is like water watercolor robot studying in a library where the query was actually different. And so a big part of calling these images is actually deciding what query to put in there. So the language model will like really really expand the query before it calls an image generation API. I don't know if you can see this anymore in chat GPT, but you used to be able to see in chat GPT when you downloaded the image that it would have like a very different caption than the caption you gave and you could kind of like latently see what the what query the the image generator sent to the generate image tool.

9:53 Then there's all all kinds of you know custom functions. So, like buy me buy me bananas, milk and coffee. so, you know, this could create a grocery cart, for you and call the like Instacart API to do this. and you can basically go as wild as you want with all these tools and and add whichever ones you want there. Okay, so there's a bunch of different ways you can think about tool calling.

10:27 the first way you can think about tool calling is just you provide a list of 50 or 70 API functions that the agent is able to inter interact with and it can call these APIs and if you do that basically what you're doing is like every step you're calling one API or if you're doing parallel tool calling you're calling multiple APIs or something like this but code is kind of a special API because code is extremely you know expressive and you can view code as like a meta tool right so within this we have like the bakers baked 200 loaves of bread how many loaves of bread did they have left here this is calling an assignment tool an assignment tool this is calling a subtraction tool this is calling a assignment tool you know etc etc so like essentially I I imagine that a lot of people have taken like a programming languages course or have some familiar familiarity with programming languages but basically you you can express each piece of code is a tree of these function calls and these the tree can also be have loops and other control flow in it. So it's essentially you know a very rich way of calling tools. and so you can also pull in external libraries. So these are now the custom APIs that you pull into your pull into your system. So you know now if you're calling pandas now you've opened up all of the like tools in pandas right and you know you can build your own utility functions. So you could call special purpose functions that you want as well. So code is kind of special. It's like one of the one of the you know special things that it's different from other tools in the way that it like opens up your ability to call tools in new ways.

12:37 And so there's actually research work that demonstrates that this is very effective. right now we basically take it for granted but this was not how not how we called tools like three or four years ago when we first started you know building agents with LLMs. And so the the standard then was like if you got a a query sorry this is very small but it basically says determine the most cost-effective country to purchase the smartphone model Kodak 1.

13:07 the countries to consider are the US Japan, Germany and India. Basically, what this would require you to do was you would call your lookup rates tool on Germany. you would call your lookup phone price tool. You would call convert and tax. Then you would call lookup rates again. Then you would call look up phone price and you would step over and over and over again in order to finally get finally get the result. But if you write code, you can just write like a single program to perform this. And so this makes it like a much richer a much more efficient way of calling tools in addition to be being richer. And so this method called called Kodak has a paper about it and some of the core results from the paper are essentially the success rate goes up and the average number of interaction turns goes down. And this is even for tasks that didn't traditionally require code. These are for tasks that were like viewed as regular tool use tasks. but like code is just like a good medium for how you would do this.

14:18 but there's a reason why people don't use code programmatic tool calling for everything. and it largely has to do with with like the level of power that you want to give to the agent. So code is very highowered. So it's expressive. It gives you loop variables and libraries. This is good. But what could be a downside of this? Any ideas? Ju just with respect to loops variables loops and variables at first maybe. Yeah. >> Get stuck.

14:49 >> Yeah, it could get stuck. So it could write an infinite loop and then you need to have something in your system to deal with infinite loops, right? That's that's a little bit annoying. what about libraries? any problem with that? >> Yeah, security problems would be one thing. Also, maybe just versioning and stuff like this, but I'm not sure if people are familiar with this, but one of the major issues with agents in cyber security right now is the agents will pull in a library that has been compromised. the agent gets compromised and then your whole system gets compromised. So, this is something something you need to be very careful about.

15:27 they're also harder con to constrain because they have the broad action space. So it's you know harder to validate and get predictable behavior. they also have higher impact. So they can you know use more disk sorry use more memory, use more disk, use more CPU and stuff like this. And so because of this we'll talk about this more in the safety lecture, but like once you move to programmatic tool calling, you need to be you need to think about sandboxes and how you're going to contain the agent and stuff like that as well.

15:59 any any questions about this? Cool. Okay. So I want to talk next about the mech mechanics of providing tools and right now every language model is basically pretty good at like calling tools but this was not you know taken for granted. I I think I mentioned this on Tuesday as well. but basically tool calls are expressed as tokens like this. And so you know a text continuation is the weather is sunny. A tool continuation is like a tool call and then you have get weather or something like this within the the tool call. And very often you have both in the same output. So if you remember react which was talked about before I can actually draw this on the board but react basically you have your text up here and then you have your tool call and then you have your tool call down here.

17:19 Yeah, this is like the end of the tool call. And so now you have the the model be being giving a response to the human while it's making the tool call. And then in here you have the like actual tool call that it's making. And so the react loop is no longer expressed as like separate code. It's expressed more as like just a single completion. if you're using one of the reasoning models that can reason like have long reasoning traces and stuff like that. In addition, you might have like the thinking tokens up here and then a message and then the tool calls. And so the thinking is like very verbose thinking that you don't show to the user. The text is the less verbose message that you show to the user and then the tool call is like the actual output you have there.

18:14 Yeah, >> the tool call tokens has to be in the vocabulary of the model during pre-training also, right? >> Yes. >> So all of these service tokens are going to be vocabulary. >> yes. So actually that's a good point. I said yes, but the answer might be no. I'm going to repeat the question first. So the question was do these tokens have to be in the vocabulary when you do pre-training? typically if you're pre-training on all of the internet, you don't necessarily have to have them in the vocabulary yet. we there are cases where you might do that but it's actually more typical to introduce them in a process of mid-training which is between it's when you're training on like moderately large amounts of data on things in the format that you like and then you might do reinforcement learning after that but you might not necessarily have them when you pre-train on the whole internet.

19:10 Cool. so I think this should likely be familiar to you know people who have implemented things in previous natural language processing or or whatever courses. But if you have the chat format like the open AAI chat format you have things like the system message, the user message, the assistant message and then these are represented as something like JSON in like in Python when you call the call the model but these get converted into something with like a special token here the system like this is a system prompt and then be concise and stuff like this. So like each chat message gets its own format and this is an example of the quen format but as I'll show later the other other functions are in different formats. So tools basically add a structure to the request and so you have a tool definition like this and this tells you that it's a function. The function name might be get weather. the parameters are city and the type is a string and this is required. And this schema gets passed to for example the chat completions function that you you would use when calling openAI or something like this.

20:44 So when the prompt contains something like this, you might have this included in the tools section of the prompt and then the output ends up being something like name get weather with the appropriate arguments like the city of Pittsburgh. So we we had a question about how you pre-train the models and you might not be pre-training but at least at mid-training and beyond you will be training the model to fit a particular tool call format. The interesting thing is every model has a different tool call format. Maybe not everyone but many of the different ones. And so Quen has like im start tool call IM end.

21:29 Mistall has tool calls and then all of the tool calls. Deepseek has their like DSML DeepS markup language function calls. They they don't call it tool calls. they call it function calls and then evoke invoke and basically you know any of these works. there might be like minor minor differences in which one gets better performance or not if you train the same model with the same data. but what you need to know is you need to know that these are different. So you can't just assume that like some parsing code that works with deepseek will suddenly work with quen and vice versa.

22:03 Yeah. >> You said you add functionality. So need to provide. >> so what do you mean by loose generalizability?

22:36 >> Yep. fine tuning set of >> Yeah. So when you're when you're fine-tuning, you will want to match like if you're fine-tuning from a model that's already been trained, you 100% want to match the tool call format. You don't want to try to get it to do something it wasn't trained on. and the good news is this is all implemented in like hugging face with the apply chat template function. And so you need to apply the chat template for the model you're using, but that's all implemented. If it's not implemented properly, you can go in and like it you ask them and they'll go in and implement it or you do it yourself or something like that. But yeah, it's it's all standardized now. But I I want to people to know that this is going on under the hood because if you don't know this, you can make mistakes and and it will not be happy.

23:29 Cool. So once you have done this the next thing is that you do tool call parsing and actually I'm going to talk about tool called kernel parsing in the next section. So not immediately here but like let's assume that if the language model outputs this you have a way of parsing it into into the function arguments. you then like have the you register the tool at initialization and you have a way to resolve it so that you get a map from the name to the tool that you want to execute. So most agentic frameworks basically will will have a way that you can add new tools. Each tool gets its own name and then you have a Python function or something like that. Python, TypeScript, whatever language. And then it it will be dispatched to that with the appropriate arguments. And so then at each time you you parse you look up the tool, you validate the action from the arguments and you execute and then you return. But at validation time, this can fail if the model gave you the wrong the wrong tool call. At execution time, this can also fail if there's like not something that was explicitly said is bad is like invalid but implicitly is invalid.

24:57 So like for example, if you have an execute Python tool, it might accept any string as Python, but then when you actually try to run the Python, it it might fail. So that that would be like an execution error. And then if it does if it well whether it whether it fails or not, then you get an observation out of it. So this is another detail that that's pretty helpful but also really annoying when you're implementing agents if you if you get it wrong. But basically each tool call is assigned an ID when you when you run it basically by either the language model inference code or the agent code and then you get a result and the result has a matching call ID and this is useful especially in the case of parallel tool calling because I'm going to talk about parallel tool calling in a bit and having the IDs matched lets you know which tool call is associated assiated with with which result. But the the problem is for instance anthropic if you have a tool call with no matched tool result it will die on you and so it will tell you this is an invalid history and I'm not going to accept this history and and generate any more outputs for you. And so in my practical experience there's times when for example the assistant makes a tool call then suddenly your your program dies or something like that it fails to write out the tool result and then you resume it it doesn't have the result the associated result and anthropic fails to fails to work for you. So this is a little bit of a a detail that you need to know.

26:47 Okay. So, any any questions there? Yeah. >> And, great question. Are the tool calls asynchronous? I'm going to talk about that in like the the next next section. Yeah. Cool. Okay. So, now now here's a very technically algorithmically difficult problem. that may be very interesting to you or maybe not not very interesting to you. But if it's not interesting to you, be glad that other people are solving it for you. so, language models are definitely not guaranteed to generate well-formed tool calls. and this is an example.

27:30 you can see that this is JSON that's missing the last closing bracket. So, this is poorly formed JSON. You put this into your JSON parsing program and it will throw an exception and you won't be able to get the the data out of it. So I I've seen a million people go in and try to like figure out if the brackets are missing and like add one post hawk, but you don't want to be doing that. That's not not fun. so there's a bunch of constraints that you could be assigning to each of the tool calls. So the first one is syntax.

28:04 So it has to be valid JSON. So this is kind of like table stakes, right? So it it needs to parse. the second thing is shape. So make sure that if you have expected or required fields, you have the required fields. the third thing is types. So you know if you have city, the city must be a string. Oh sorry. Yeah. yeah. So if you have a city, city is a string. if you have values, you know the units are in the appropriate units. So this can all be expressed through something called JSON schema or basically any schema library. so it's like this is a city the type is string and the units can be Celsius or Fahrenheit and city is required.

28:54 So the the good thing about this is this has machine readable validation rules. So you can check if like a call follows the schema. But one way that you can express whether the schema is valid or invalid is by defining a grammar over over the schema. And how many people did like contextf free grammarss or like context or those sorts of things? Regular grammarss. How many people know regular expressions?

29:28 Okay. A lot of a lot of people. Okay. So a regular expression is a regular grammar. does anybody know how you can parse a regular expression? >> Finite state automaton. Yeah, that's exactly correct. so a finite state automaton for for those who are less familiar with this is something that looks a little bit like this.

30:11 so you have a state and every time you take a token as input, you move to a different state. So if you get A, you would move to this state. If you get B or C, you move to this state. And This this works on something called regular grammarss. So regular expressions like the the slashes and dot stars and stuff like this that you might use to search for search text are reg are regular expressions and they can be expressed through regular grammars.

30:47 JSON schema cannot be expressed through a regular grammar. it can be expressed through something called a contextf free grammar and you can the context free grammar can basically make a tree that looks a little bit like this. So every JSON expression can have brackets on the side. It can have commas between its members. this can be a string and this can be a unit. And if the JSON schema is in the correct format, you should be able to create a tree that looks like this.

31:21 If the JSON schema is in the wrong format, you might create a be able to create create most of a tree. But then when you get to a unit, a unit cannot be K. So basically this would this would fail even though a unit can be K, but we we defined this to not accept Kelvin as our our unit of temperature. So basically if you can parse the tool call with a contextf free grammar defined from your JSON schema you can you can do this and I'm not going to go into a lot of details but a finite state automaton cannot parse a a contextf free grammar but there's something called a push down automaton that can allow you to do this. So basically it adds a stack to the regular grammar and you push things onto the stack pop things off of the stack and eventually can eventually can decide whether it's parsible or not. And so what you do is you start out at the very beginning with a state in this push down automaton and you step through and validate your output as you as you generate it from left to right and I'm giving some examples from X grammar and X grammar is a thing that's actually pretty widely used in all of the LM inference libraries. It's actually developed by people in the machine learning department here. so what you what this means is every time you get an LLM predict the logits of the next token, you take a look at your grammar and if the grammar says this is a valid next token, you get a score of one and if you have an invalid next token, you get a score of zero. And you set all of the invalid next tokens to negative infinity. for the output logits and then you renormalize so that you get probability only on the valid tokens. And so what this lets you do is this ensures that you generate valid output.

33:31 and so this is another figure from the Xgrammer paper and they do these tricky things where basically there are some vocabulary that are always valid or always invalid and you premputee these and then there what there are those that are valid only in some contexts only in some stack context and then you calculate them on the fly. So it's very interesting algorithmically if you like algorithmic stuff. If you don't like it be glad that the people in the machine learning department are doing it for you basically. so because of this almost always if you are if you are have this enabled you will and set your appropriate JSON schema you will get wellformed to tool calls. So that that's that's great.

34:27 There's one case where you actually can't get well-formed tool calls despite this. may maybe I can give a quick quiz. This is very very difficult. So I don't know if anybody can get it. But any anybody have an idea where even this would not be not be enough. Need to be familiar with with LMS I guess. >> Yeah.

34:58 I I think that's close enough to what I I wanted to say. So basically when you run out of tokens, so models can only generate so many tokens and you might run into a place where you're still on a valid path through the automaton, but you're not at the final state. So you generate like part of a JSON output. So very very good. and I I've actually encountered this. There's theoretically a way you could do that which is like you count the number of output tokens that you still have left and you you cut them off but that's if if somebody wants to implement that in next grammar as a extra credit assignment you know be happy to happy to have that okay so now let's go to rest API calling so there's two different ways to there's different ways to Express tools and I'd say probably most people are familiar with REST APIs. The these are the ways you interact with web services. but actually sorry there's one other way to call APIs and one way to call APIs is through programmatic tool calling where basically you just give the model a Python program and you say use the stuff in this Python program to call tools.

36:14 And so if you do something like that you might have like weather lib.py pi and you have the get weather function in here with its city and its units and then you just say call this function whenever you need to and you give it the ability to execute Python and it will be able to call that tool. So in programmatic tool calling this this works but this is not callable remotely and it's also not callable from the like standard method of tool calling that you use that you use when you implement this not through non-programmatic tool calling.

36:52 So there's a way to turn this into REST APIs. Has anybody used fast API before? It's a pretty common Okay. a lot of people. It's a pretty common library. And basically what this allows you to do is this allows you to create a web web backend extremely simply by just setting up a fast API giving an API key so you can get people to validate against the API and then writing a Python decorator where weather becomes becomes this. And so this is good for a couple reasons.

37:29 The first reason is maybe if you don't want people looking at the weather then you can prevent them from looking at the weather. But probably more importantly if you don't want people like getting your personal data or something like that you can require an API key and this will allow you to block anybody who doesn't have an appropriate API key for instance. the other reason why is this gives you a JSON schema for free. and when I say for free of course you know this is something that's implemented in in fast API but your entire API can be expressed as a JSON schema and that JSON schema can then be provided as is to a language model and that language model can then use it to you know specify which calls are allowed and so then you could go in and process this and whenever the agent made a tool call you could then send it to your API server and they it could use that on the API server another option for calling these is you can just use the curl command. And the curl command basically allows you to call these APIs directly like this as well. And so if you have a a coding agent that has access to bash, it has the ability to to call like this.

38:44 So this is maybe maybe reasonably straightforward if you're familiar with REST and stuff like this. Any any questions or comments or If you haven't played around with fast API, it's nice. It's very easy to use. But the reason why I wanted to cover this first is this is kind of like the traditional way of of programming you know before agents. there's something called MCP the model context protocol and this was introduced by anthropic maybe twoish years ago or a year and a half ago and the idea was this is was designed to be a standard way to give your model additional context.

39:31 When I first saw this I actually was like why do we need to do this? I don't understand why we need to do this. Let's let's just let our agents call APIs. but there is one important point about one one or two important points that make MCP different from just calling a normal API. and I'll I'll explain them here. But first I'll explain why I didn't real I didn't like I thought this was not actually necessary. And basically the reason why is every time you set up an MCP, what you do is you have your AI application and then you set up a couple MCP clients. So like maybe this is your weather client your this is your weather MCP and your your GitHub MCP or something like that. So you can ask what the weather is what the weather is and push it to your GitHub. and the reason why I didn't think this was necessary originally was because like why do you need to run a program to call an API, right? We already have good ways to call APIs. We just, you know, send it to the the REST API server.

40:43 But okay. Yeah. So so that's a little bit of an aside. I'll I'll get back to that in a moment, but for a little bit of an aside, there's a really nice library called fast MCP. And fast MCP is basically designed to be fast API for MCP. So if you want to create an MCP server for whatever you want, you can just like basically swap out fast MCP and swap it swap out fast API and swap in fast MCP and you can create an MCP server very easily.

41:16 you can also take in a spec that was generated by like a REST API and just turn it in into an MCP as well. But here here's the main reason why you might want to use MCP as opposed to an API. and the reason why is because it gives you an extra layer of security. And so MCPs have an MCP API key. And this MCP API key is something that you need to provide to the process that's running your agent.

41:53 Separately from this, you have the upstream API key. So like let's say you are getting getting your agent to connect to your GitHub. The upstream API key would be your GitHub token and you would provide that to the MCP server, but you would not provide it to the agent. And then what you provide to the agent is the MCP API key, which allows you to authenticate into the MCP server. And so the reason why this is important is what happens if you give your agent the GitHub your GitHub token and your agent suddenly decides that it's a good idea to push that to your public GitHub repository.

42:28 Not good, right? Your account gets compromised and you get malware installed onto all your repositories or something like this. So you want to give the the agent something that is relatively harmless. so that even if it leaks it for whatever reason like that leak is relatively harmless and then the the actual credentials get get preserved and so that was basically the reasoning behind it. So just to go into a summary, they all give you the ability to add a set of APIs into into your agent. Like you could do that through a regular JSON schema or through an MCP.

43:09 the but a lot of other things are different. So like for discovery you might need to fetch an open API document. For MCP there is a special API that allows you to list all of the tools. for invocation you call an HTTP thing here for MCP there's an MCP transport and this can be through various ways. You connect through standard IO or you connect through an H a socket or something like that.

43:39 I already talked about authentication and MCP actually has a few other has a few other like conveniences like you can add resources, prompts and extensions, but I'm not going to go into a lot of detail about that. Cool. actually, there's two other things I'd like to mention. there I don't want to I I guess I forgot to add them to my slide. so the first thing is MCP registry. There's an official MCP registry. So if you want to find MCPS that allow you to do something like this, you can, you know, go to this registry and find any of a very large number of MCPS. There's also like a million and a half MCP server like websites that you can go to that are are ranked based on popularity and stuff like this. Another thing is I've been talking a lot about the Pittsburgh weather and the funny thing is today I prepared all of these slides like earlier of course but today we actually got Pittsburgh weather that was simultaneously rainy and sunny at the same time. And so I managed to break my own API because it cannot return multiple states at the same time. But yeah, it was quite the coincidence. You can see the torrential downpour and the the bright side at the same time. Okay. back back to serious stuff.

45:16 Cool. so another big kind of like development pretty recently is parallel tool calling. in parallel tool calling. the the basic problem we want to solve is the same problem that I talked about when I talked about like the codec and the programmatic tool calling which is if you call tools one at a time. the it it could become very long for even a simple operation because you might do a weather and then do a calendar and then do a flight. but if tool calls are independent, you can actually call them at the same time. And this is done pretty simply. You just like add add multiple tool call blocks to a single agent response.

46:04 The there's a few nuances about this. The first one is you can only paralyze tool calls if they don't have dependencies on each other. So, like if you were looking up the weather for Pittsburgh and reading today's calendar and looking up flight status, that might be, might be good. but if you need to find a customer ID, use it to fetch orders and refund the selected order. Obviously, you can't do that. So, there's limits to the parallel tool calling you could do. And if your calendar or your flight was dependent on the weather, then you know, obviously that'd be a problem, too.

46:37 So, there was a question about this. do you execute the tool calls in parallel? And the answer is typically yes. You can just use typical Python syntax to you know have a whole bunch of tool calls and then execute them and then use something like async.io.gather to to return the results of them when they run in the same time. So this is this is pretty common. Another thing I'd like to point out is this is a major difference between really efficient language models in note efficient language models. And there there's this weird weird paradox in agentic language models nowadays that more expensive models more expensive models can actually be cheaper if you measure them on a taskbytask basis. And there's two major reasons for this.

47:33 The first major reason is more expensive models can be smarter. So they pick like an appropriate solution more quickly. but another thing is more recent models do parallel tool calling a lot better. So they make a whole bunch of calls at the same time. So if you're using a coding agent or something like this, you use the the better you use the better models and they are doing like a bunch of view images at the same time or they're like writing a bunch of different files at the same time or something like that. That's because they're doing parallel tool calling.

48:09 Well, does anyone have an idea of like why they got so much better at this? They like this was not a huge thing even though parallel tool calling mechanisms have existed for a long time. This was not a huge thing until maybe like six months ago or something. Yeah. >> Reinforcement learning just forced them to do that. >> Yes. Exactly. So reinforcement learning kind of forced them to do that and so the models were very heavily incentivized to fish finish tasks quickly and parallel tool calling is one of the good ways to do that. So if you do reinforcement learning with a very heavy like brevity penalty on on how long the task execution is, you can get like a lot more parallel tool calling.

48:53 Cool. So the final thing I'd like to talk about in this lecture today is evaluating tool use. And so I'm not going to talk about evaluating endto-end agentic tasks immediately yet. but rather talk about evaluating just the tool usability of language models because it's kind of like a prerequisite for language models to be good at good at agentic tasks. And there's a bunch of data sets for this. The most famous one is the Berkeley function calling leaderboard and this has four versions. it started out with version one which was basically single turn tool calling. it's then evolved through the four versions and now it covers multi-turn tool calling. It also does have aentic tool calling but this is more not for like endto-end coding agents or something like that.

49:48 It's tool calling agents and they also have tool calling robustness. So they measure hallucination of arguments. They also measure format sensitivity. So when I talked about like the all the different formats they have parameters about that as well. So when you go in to evaluate the whole stack there's a number of things that you want to evaluate. So the first thing is did the language model choose the appropriate tool for the task. So it might have picked like a an inappropriate tool that can't allow you to do that the task at all.

50:30 the other thing is arguments which is you know whether you you add the right values. if you're talking about agentic tool calling you can talk about whether it did it in the right order and then you can measure endto-end task accuracy for if you're doing agentic tool calling. And all of these are kind of metrics in this leaderboard. Then separately from that there's also efficiency. So like how quickly can you solve the task? how many tokens did it take? How how much did it cost? reliability. So you know if a tool call times out does it retry appropriately? If a tool is out of order, can it choose a a different one?

51:15 And also safety. So if you have adversarial outputs where you ask it to use a tool in an inappropriate way, can you do this? And all of these are like over the four iterations of this benchmark. It covers all of these dimensions. yeah, I kind of covered that already actually. So one other really interesting benchmark for tool calling is this open router tool call air rate benchmark. I don't know if anybody like uses open router or has seen this before. Maybe no.

51:51 Okay. So this is for the same model. All of these are for I think this is GLM 5.3. So it's like the same the same model and it's measuring the tool call error rate simply the only thing they varied is who serves the model to you. And this is this is pretty important obviously because you think oh I'm going to use this model maybe you know it's going to be the same regardless of who provides it to me but actually the difference is between 15% and like 0.01 to 0.05%.

52:30 Any idea why this might be? There were some there were some hints in the lecture and some not some hints in the lecture >> is decoding algorithms different maybe someone's using like cheaper version of the model or quantized version of the model. >> Yeah, great great points. So I'll I'll repeat it. So number one is the decoding or the inference algorithm could be different and one very big difference is whether they're using quantization of the model or not or like what level of quantization of the model they're using.

53:09 So my my personal experience is FP8 models FP8 is kind of like the default level of quantization for the model get a lot fewer mistaken tool calls than FP4 models because FP4 is like compressing it very heavily. It's saving money for the inference provider, but it's it's usually a worse model. There aren't very many FP4 quantizations that can allow you to get the same performance. Another one was speculative decoding. I don't think we're going to talk about speculative decoding a huge amount in this class.

53:45 Maybe I'll talk about it during long context or actually may there are a few algorithms specifically for agents which are interesting. So maybe we can talk about it. But basically what it is is you have another cheaper model that predicts what the more expensive model is going to do and then you use that to like speed up your inference. spec lossless speculative decoding probably shouldn't make a difference because it it's like not changing the underlying result of the model. But if they're doing some sort of like lossy speculative decoding then that might might cause it to be different. But then you know calling it the same model is also probably a bad a bad idea.

54:25 there's one other thing that I mentioned and anybody have an idea this was actually mentioned in the in the class today. What can you do to make your tool calls more successful? >> Yeah. >> Constraint decoding. Yeah, exactly. So these different providers might be using different constraint decoding algorithms. just practically many many of these providers will be implementing their own inference algorithm. So they'll have their own inference algorithm completely implemented from scratch. some of the providers will be using an open source one like VLM or SGLANG. And VLM or SG lang support different grammar-based decoding algorithms. And so some of these providers just might not have that implemented and so they're entirely relying on the language model to do good tool calls. otherwise they might not. A final kind of like boring reason why tool calls might fail is the inference provider might be unreliable.

55:37 So it might go down some of the time and then the tool call would fail. Cool. so these are the that that's the main thing that I I had for today. so tool use is basically a layered system. you know the tools add capabilities such as retrieval, execution, generation, application. They're what make an agent an agent. but we can't get that for free. And you can add schemas to the prompt.

56:08 the models emit calls and then you validate dispatch and match results by ID. And we can add constraints like grammar-based decoding. they don't guarantee semantic correctness. So if you have like a Python program, the Python program might still be wrong even if you generated you know a string that could be parsed as Python. and there's a number of ways you can interface with them like direct rest calls and MCPS. and then we have a lot of systems that orchestrate dependencies and concurrency and allow us to evaluate.

56:43 So yeah, that that's what I wanted to talk about for tool calling. this will be something that is covered in the harness assignment or the first assignment where you build your agent. So you'll need to need to deal with this. but and next time we're going to talk about context management for long context in long context LMS. But any questions to wrap up? Okay. Yeah. Yeah. And I'm going to repeat in case people didn't hear in the back, but the first assignment for reflecting on the lecture and talking about something you learned in the lecture is going to be up on Canvas. So, make sure you make sure you submit it. Thank you.