Transcript
0:05 Today our topic is going to be around the two years. With that, let's go to our content today. First, I'm going to wrap up a rag that we covered the last week in 5 minutes and then we are going to move towards two years particular around the why we need to consider two years for agents. What are tools and how to get a model to learn what type of tool to use. And then how can we actually scale to many many different type of tools we have.
0:39 We will then shift from how to teach model to do it to how to have a standard that people can use different type of tools from a standard interface aspect where we'll cover rest, function calling and MCP. The last part of this lecture will cover many different type of function call and the two years data sets and evaluations. So hopefully by the end of this lecture you understand why we need tools, how model learn to call them and how tools are exposed through standard interface and how to use is validated.
1:17 So with that, let's coming back to what we were talking about the last week. So I talked about the rag. Rag is more from a design principle perspective where in the process of answering a question or working on a task, we can actually do retrieval and then use those retrieved content to better facilitate the current task. So that's rag and in addition to many of the algorithm we have learned, I want you all to pay attention to issues with rag because this is where rag actually fail.
1:54 On the retrieval side, as we mentioned, maybe agents actually some information which they should not have access to. So, that's called a permission leak. On the reader side, there is this phenomenon called a lost in the middle, which I will cover in the next few page. distraction. Agents get distracted by very similar but irrelevant content. The real world also has a lot of like conflicting knowledge. So, agents may, fetch information that cover different aspects. For queries such as, is drinking milk good for my health? You may get articles that talk about the benefits. There are also a lot of articles talk about, drawbacks. So, how would the agent actually take care of this conflicting evidence? It's actually a big challenge.
2:47 On the loop side, so, if you think about the rag, it's like retrieve and then read and then generate. So, this is actually a compound system. some of the errors or mistakes agents made earlier may also lead to, issues in the later stage. And then, more importantly, we will cover this detailedly later this quarter, is around the prompt injection. That is, what if, bad actors actually decided to inject something, negative or malicious in the retrieved documents. And this actually will destroy or influence your rag system.
3:27 So, I want to go deep into one part here called the lost in the middle, which is actually a very, big issue rag used to suffer from. So, let's get it started. so, this is a very simple setup. if you read this example, the prompt basically says that, let's write a high-quality answer for the given question using only the provided search results. And then, what we are going to do is that we give you many documents here.
3:58 Document one, document two, document three. And then, the question is supposed to read all these content and then retrieve the right document and then do the question answering. So here you can see that a document two is correct as I bolded them here. So the question now is if this is how we do it, why don't we just give a question answering system a lot a lot of documents? We can literally give the internet here if the context window permits.
4:31 And also what happens if we have instead of three but many many many documents? And also what if the position of the correct document is not the two, it's actually 13, 43 or just like somewhere in the sequence? Would that change the rag results? So the answer is yes, and this is a very kind of classic lost in the middle results. So if you look at this trend, many of them is more like this kind of like U-shape.
5:05 Where if the correct document is in the top or in the in the most recent or the last position of your sequence, models are very likely to use them well. But if the correct document is somewhere in the middle of the sequence, you actually get worse the performance here. So this is actually very surprising. This is also what we call a position bias when we think about rag. In the bottom I show you some numbers. So basically if you just without a rag, you just do close book QA, GPT-3 turbo here got a result like 56.
5:45 And then if you could have given them the relevant document, you can actually boost this number to 88, which is pretty high. This is why rag is useful. But then if you look at the in the middle part, we actually really just like degraded to even without a search. So this is quite a surprising and the answer the correct answer is actually in the context. But models actually ignore it. So, this is the very classic lost-in-the-middle phenomena.
6:20 so, the same paper, but I'm going to instead of thinking about the positions, let's look at how many documents we can retrieve and how does that affect the results. So, the yellow line here is the system the authors built, that is how often the answer is somewhere in the retrieved set. And you can see that it is slowly grow to about 90% at around the 50 documents here. And I think the other lines here are reader accuracy.
6:53 most of the reader actually saturates after 10 to 20 documents. And around the like 50 to 60% here, and some even like go down here. So, the gap between the yellow and the rest is the evidence that models actually cannot use. So, more retrieved documents do not mean better results. So, you need to pick this kind of like K carefully and then re-rank etc. So, all of this is trying to share with you that even for a algorithm or design principle as simple as rag, it may suffer from certain issues such as lost-in-the-middle or such as they cannot actually use all the retrieved documents there.
7:40 If we go this a little bit deeper, the reason why people like rag is earlier days as model hallucinates, we want the evidence, we want the citations. if they tell us that coffee is not good for our health, they may need to give us the article that supports that. This is a mechanism called a citation, and this is actually a earlier approach to deal with hallucination. But the interesting part here is that because citation is also generated by LLMs, citation itself can also be hallucinated. That is, you were supposed to get a citation to support your answer, but your citation itself is also wrong. This is why you probably have heard the lawyer situation where they use LLMs to write a court case where they were presented with very convincing instance, but it turned out to be a hallucination.
8:41 if you kind of quantify this from a rigorous perspective, so on my screen those two tables actually are are from the original Lost in the Middle paper. And they actually look at the different search engines being Perplexity and UChat here. If you look at the perceived utility from users from a 1 to 5, it's actually pretty high. 4.5, 4.48. So, users think of the results from RAG of high quality and fluent.
9:17 If you look at the citation correctness here, so the left side is the citation precision and then right side is the citation recall. So, this is a number from 1 to 100. You can see that well, the precision is only 75, which means that the other 25, other one quarter is actually fake or hallucinated content. Which means that if you get a four reference there, one of them is wrong. And same is if you look at the recall.
9:48 This is also pretty like low here. and I think that this actually reveal a big issue of a lot of the RAG. that is, even we can get the right content, the generation may still indicates issues. this is also a phenomenon that I want to connect with research community. So, LLM citations and hallucinations is actually a big topic. last year, a team was looking at NeurIPS, which is a big machine learning conference. A lot of the papers like getting accepted there, and they check how many of those papers actually contain hallucinated citations.
10:33 And turns out that there was at least 100 new hallucinations in NeurIPS proceedings. And this is the paper many of us read to get to know algorithms, to know a field, but we start to see this. Not only with research papers, we also start to see this hallucination of citations in student research project and course project. So, make sure whatever you submit represents your standard and the quality. this year, I clear conference, which I served as a program chair, we also encounter a lot a lot of hallucinated reference.
11:11 so, as a result, there was a conference level action to do desk reject based on hallucinated reference. So, all of this is just to let you know that those content may look very authoritative, may have a citation or link, but it's a little bit unclear whether they are correct. So, the last piece of information I want to present here, because this is a really neat algorithm. It's called a needle in a haystack here.
11:43 so, if you look at this figure, the x-axis is actually the context length. So, here it's a range from 1 to 200 K contexts. the model being tested here is a very earlier model, Cloud 2.1. And then you can see on the y-axis, imagine we have one correct fact, one just like one, one correct fact in a long document. And then, what if we are going to vary the document lengths along the x-axis, where the facts actually sits somewhere in the middle? And then, we can ask it for that from a QA perspective.
12:22 A good rag system is supposed to read all the documents and tell me the correct answer. And then, I think that the interesting thing happens is that as the context gets longer, you can see that the first from a color perspective, we seem to have more red color here. Redder means the model got it wrong. Green means that the model got the results. And then, you can see that shorter context here are almost green. As the context get along, red spreads. And most of those actually focused on the top or the middle here. The bottom seems to be pretty good. That is, as long as the correct answer is what I just read, then I can get it right. But if it happens like several decades ago or like a several document away from me, I cannot get it right. So, this is actually very surprising because for the first the time, we see that despite the model can handle a 200k token context, it cannot reliably use it.
13:28 So, if we do the same kind of a needle in a haystack test with GPT-4, this is actually a very performative model. and here, the context window is 128k here. So, first, we see that seems great because the color we see here is mostly green here. but you also see that we there are some kind of degrade here, especially around like 73k, we start to see some kind of a red yellow color here.
14:01 and if you look at this box that we highlighted, I think most of those facts, they are around like 10 to 15% during the during that range. And same story as before, the facts at the top and the bottom survive. But in the middle at the longer context is actually where this kind of rag systems fail. So, it is the same story. Lost in the middle. Okay. So, this is a quick overview of many of the issues with rag. Despite that, I think it's still a very cool design principle because it gave model the capability to use external knowledge that's not memorized by the model. So, still a very good principle for us to think about when building agents.
14:51 Any quick questions before we move to to use? >> Sorry, I might have missed it, but is there any intuition for why that happens? >> Yes. So, I think there several reasons if you think about this. First is that from like if you think about it from like attention perspective, especially when some information is in the very middle or just like in the in the like sorry, in a earlier stage of the sequence, it's just a little bit harder to take care of this kind of long-range dependency.
15:34 second, I think it's that this seems to be a artifact or a position bias that we have observed that maybe when there are a lot of information just as shortcut that humans may make, we tend to put put a lot of attention on what happened yesterday, but tend to forget what happened earlier last week. So, you still to you seem to observe this. And I think this is indeed indication that models cannot actually reliably use the context. We'll see more and more of those when we go to the memory component as well.
16:12 Yes. >> So, if you want to like retrieve external knowledge, is there any alternative to rag or like you have to use rag? >> I didn't hear your first sentence. If you have to >> If you have to use external knowledge, is there any alternative? >> Yes. So, this is what I was trying to summarize. So, I showed you a artifact or issue with RAG, but that does not mean that the RAG is not good on its own. So, I think as a overall general principle, especially if you have external knowledge, doing like retrieval, fact sheet. This is actually one type of tool use as well. So, retrieval in itself is also a tool use.
16:53 so, it's still a very powerful approach, and when you need external data, I would still recommend that to go ahead with RAG. However, just as large language model being a very interesting technology, there are promises of it. There are also issues that we need to pay attention to. So, what I would recommend is that the for your task, you go ahead with RAG, and then you do a lot of the validations, or just like validity check, just to make sure that, okay, I do not suffer from this, and the sense seems to be not only correct, but also I'm not influenced by this kind of position bias, or in the process, all the retrieved content are secure, and there's no like bad actors injection of bad content.
17:40 Okay, with that, let's continue. so, we need to talk about tools. And as I just mentioned, retrieval is also one type of tool use. before that, I want to give you more examples of of earlier days. This is again around the GPT-4 stage. So, at that time, if you ask questions such as, "Who is the current president of United States?" the model will be like, "I don't know." if you ask them like, "What day of the week is it today?" it cannot help you answer that. Despite that they can solve very hard math problems.
18:19 something interesting occurred, that is as Professor Yejin Choi mentioned in her talk, when it comes to multiplication. So here, let's just ask GPT-4 a very simple question. multiply 999 by 867. And then you get a correct answer. Sorry, you get a answer here, but it's not a correct. so overall you start to notice that it's so interesting like this this very capable technology, but then their accuracy for multiplication is only 59% accuracy.
18:55 Their their performance on AMI or other kind of score is probably much higher than this. If you argue like, "Okay, maybe GPT-4 was not a very capable." this figure is using O1, which is a pretty strong reasoning model that's supposed to be much smarter and capable with math. So let's do the same task. But then you also see this trend. When we increase the number of digits in the multiplication, even when it's only like four, you start to observe that things start to get red here. Redder means like failure.
19:32 So you see that we start to have worse and worse performance in multiplication. Multiplication is not a harder task. I mean, as long as you learn how to do A * B, you're already humans can actually do the generalization to multiple digits easily, but this seems to be an issue with with earlier models. And this trend is actually very general. Like here on my screen, similar web, you see that not only for multiplication, for puzzles like solving puzzles, or even doing like a dynamic programming, as the situation get more complex, model performance start to get worse. That is maybe models only know how to solve easy problems, not harder problems.
20:21 This is not a bug in one particular model, and I think of the pattern here is that those models seems pretty good when the problem is small. And then the accuracy actually drop as the number of steps that it needs to be used actually grow here. So, this is actually maybe a component that what if we can actually delegate that to someone else? Because the model seems still capable. They just cannot handle very complicated task.
20:53 this actually lead me to the introduction of tools here. So, here I summarize why LLMs are its own is actually not a good enough and why we actually needed to use. first, as I showed you, for this kind of relatively longer sequence multiplication, models seem to struggle. And this is true with like multi-step logic deduction, exact or very verifiable settings.
21:23 The other thing related to like what's the weather today in like in Alto, if the model didn't see that in the training coppers, it's really hard for them to get them right. So, you can imagine that the long tail effects or post cutoff events or private, sorry, private proprietary or external data, models seem to struggle with it. the other thing is that models seems to do well with text, code, math, but we also have a lot of like APIs, database, like non- language interface.
22:02 we also have the physical world. We may also have some kind of like live states that we want to get out of. So, this is actually what agents can do for you. I mean, if they cannot do like three digits by three digits, maybe we can give them a calculator here. If they cannot write dynamic programming easily, maybe we can get them to write a code instead. we can also give them symbolic solvers, verifier that we have from other research field.
22:34 In terms of content that they never see in the training coppers, we can give them retrieval, rag, web search. We can let them access newsfeed, today's news, live APIs. They can also do some kind of rag over like company specific coppers. This is more like the internal coppers here. for a lot of like a non-language interface, maybe we can enable agents to use function calls, APIs, MCPs. They can use browsers. They can they can work with robots, etc.
23:10 So, this is why like having tools is going to really enable agents to do a lot on top of LLMs. I think in earlier days, there was a lot of debate on what a really defines an agent, what is agentic system, and what is not. And one big factor is actually whether the system knows how to use tools. And this is actually a really big a differentiator there. Okay, so what what are tools?
23:41 we do not need a formal definition here, but I just to kind of separate this with a lot of the real world tools we have, such as a car. here, we are talking about the large language model to use. you can view with them as a function interface to a computer program that runs externally to the LLM, where the LLM actually generate the function calls and input the parameters, arguments in order to use the tool.
24:12 This is probably a little bit abstract. And I want to mention two words here. first, external. For a lot of the tools used, the model does not need to do the task themselves. It only needs to write down the tool use kind of arguments and then some external program will do it. So this is why it's like external. We will see some of the examples later. second word that I want to highlight here is function interface. So a lot of the two we are going to see is not like okay here's a car.
24:47 what what we are going to see is that it's a text stream with a name with a signature with some kind of like parameters or arguments. So this is actually what do we want to model those two learn to use. It's actually a very interesting set up because LLMs in general are trained to deal with text. So if we could have put the two use also in similar textual formats, maybe it would work better. So this figure you can see that one of the two use, if we want to know what's the time right now, they can actually use get a time to access the real world and figure out that it's 3:00 p.m. right now.
25:26 And then similarly, they can do some calculations instead of doing this very complicated math that's going to fail, it give it to a external calculator here and then can get it the correct result. So this table actually give you a a wider range of examples in terms of the two use. I already talked about like knowledge access or computation activities. They can also interact with the real world such as get the weather, get the location, look at your calendar and use your email.
26:03 they can deal with non-text non-text modalities such as maybe they will call image generation tool to generate a meme for your specific task. So otherwise you are going to rely on the model themselves to do image generation, but now you can actually delegate it to external image generation tool. so the last column here is actually Samsung I think open to use to a wider range of domains.
26:34 Not only this kind of very ready-to-use tools such as calculator or like get the weather API. We also have models that we have trained in the past. We may have really good a machine translation systems. Maybe there's a dictionary that is someone maintains and it's just awesome to explain the meaning of a word in your favorite language. And then a lot of those external tools or systems models that we have built, they can also be used as a tool here. So the definition of tool is actually very broad.
27:09 Okay. So again, let's look at the one last example here before I go deep into tool use. So our familiar multiplication example here. What is 17,423 * 3,892? I do not know the answer, but models also do not. However, they can actually write it down in this format. So the one that I highlighted here in yellow is actually a specific tool call here. So you can see that it's not like some other format. It's actually in text. It uses like a specific indicator to let the models know that here you need to call a calculator here to calculate this kind of digits multiplication.
28:00 And the same on the right. So you can see that how's the weather today? It's actually pretty hot. And then the model is going to call check weather and then look at the some external dashboard or database and then tells us that oh, it's sunny today. So this is how they do the tool call and then internally it's all this kind of like arguments or string here. Okay. So before we talk about how to really enable models to learn how to use tools.
28:33 Any questions so far? >> So maybe a decade ago we had all these interface definition languages and all the distributed object stuff, all about type checking in order to make sure that we could get an interface call right. And it seems like all your examples were strings. So now we've blown away type checking and now we're to semantic questions. So how do you deal with versioning of the semantics? Is that all just both sides have to make it right? It doesn't seem like it's caller makes right, it doesn't seem like it's callee makes right.
29:09 How do you deal with that? >> Yeah, great question. So we will briefly hint about that when we come to the standard interface as well. The short answer is that models suffer from that. So you can imagine because you rely on AI to generate a string with that is specified input arguments, functions, things like that. It's very likely since they can already hallucinate the citations, they can also hallucinate the data string, which means that if you're going to do external type check, things like that, they fail. So this is actually a bigger failure of tool use, which we are going to cover later.
29:45 Yeah. >> so when it calls functions, would it also be possible that it sells like Python code and then creates like these children cells? >> Yeah. So good question. In case I didn't hear you clearly, I think you are asking whether the models actually can come up with tools themselves on the fly. The answer is yes. So in addition to a lot of the widely used tools that I'm going to share, you can also imagine if the model is intelligent enough, they realize that in order to do this task, it will be much easier if I generate an interface so that I can just do everything on the interface. Then that's actually a generated tool on the fly. This is actually very possible and there has been research showing that.
30:31 But again, since it's going to be a new tool, right? You suffer from all the issues that I talk about it such as hallucinations or maybe this too is actually not the generated by the model but the model memorize someone else too and then as a result give it a here to you. Okay, we are going to continue. In the next 15 minutes we are going to learn a little bit of math. So I want to give a warning first. So the question that I want to share here is so far I tell you why we need a tools and then what a tool is. So if you think about it from a model perspective how does the model know when to use tool? How does the model know what kind of tool to use?
31:16 It feels like a lot of decisions we need to make. And I think many of us have learned the machine learning. If you figure out that there is a question that we need to like decide whether to do it, which which tool to use. This is actually a very classic kind of machine learning question. So what I'm going to show next is let's teach models to use tools. So the title here is literally what I have been saying that is large language large language models they can teach themselves to use tools.
31:49 So let's be a little bit of slow here. The system I'm going to share with you is called a tool former earlier from meta. It basically trains large language model to decide which API tool to use, when to use them, what arguments I'm going to generate to pass and how to better incorporate those results. So we have four stage here. On the right this is actually what a tool call look like in earlier days.
32:21 So I'll just use the a shorter example here. So out of the 14,000 participants 500 sorry 400 or passed the test. So here in order for the model to know that the next token is 29%. They actually need to do some calculation. So, what's happening in a hypothetical string is that we now insert this this like two arguments string here so that the model knows that here I need to do a calculation of 400 divided by 1,400 here, which is 29, and then they use the 29 as the next token.
33:06 And similarly here, if you look at the first one, the New England Journal of Medicine is a registered trademark of So, at this stage, it feels like a good timing for models to use a QA system. The QA system is going to ask, "Who is the publisher?" And then you get a like Massachusetts Medical Society. And then the model generates the MMS here. So, like now let's figure out when should I ask this question? Why is it here, not here?
33:39 And then why is the QA not a translation? So, those are the questions we need a models to learn. Okay, so before I talk about what's going to happen, I'll share you that in two formers, the first system that they kind of learn how to use tools, or maybe the first of many systems, actually only work with five tools. Today, our agents can actually use thousands of tools. But this is actually a earlier idea. So, there was only five systems we provide. A QA system, a cal- calcu- calculator here, a search engine, a translation system, a calendar. And this is all what we have.
34:20 Okay, only five tools. So, now think about a method. So, what would it be a good training? I already showed you some of the examples earlier, such as this like very slightly different format that very different than natural language, but it's also certain temp- type of thing that's related to natural language. So, what if we can actually create a data set? The data set is going to be that a kind of string that I showed you. So, every real sentence will have some kind of augmented format with API calls. And then, what if we can actually train a model on top of that? Then, at inference stage, my model knows when to generate the such tokens, when to how to parse the results. So, we literally turn that into a learning problem here.
35:17 On my screen, you probably saw that I showed you this like this is a very earlier model called the GPT-J here. so, we are going to kind of fine-tune a small model. Now, I'm going to show you how we actually decide how to really train this kind of system. So, take a many step back. What I want to have is I want to have a regular text string. And I want to insert a few two calls in this string that are making it look like the format I shared earlier.
35:47 And then, I'm going to train a model on top of that. In the inference stage, whenever they saw this kind of prefix, they are going to generate a two call at a certain position. So, this is this is literally a learning problem. So, as I mentioned, there are many steps. So, let's now kind of like go a little bit of deeper here. First is a sentence has so many words. Like this sentence, Pittsburgh is known as the Steel City here. So, if you need to insert a API call or if you know where to insert the API call, where is actually a design choice.
36:26 So, we are going to call this magic position I here. So, for any kind of input, we first need to select that position I where if we insert a API call there, that it makes a lot of sense. So, assume we know this. This is actually how we pick where to insert that API call. And where is actually determined by this probability here that is PI equals everything so far. And then the probability of I insert an API here. So we basically want to find those positions that can give us the highest probability of inserting an API call.
37:06 So this is the first step. Once I decide that okay, this is a position I want to insert an API call. What should I do next? I'm going to just like sample different API call I can use here. Because once I know the position is three, I could have used a QA system. I could have used a math. I could have used a search engine. I can use many many choices here. So which call is going to be great?
37:32 So here we will have several candidates of API call. And then we literally are going to execute those API calls. On the figure here you can see that while I know that I need a QA call here, I got the answer that it's called a steel city. And then I execute that and I get the answer. So all the execution actually happened externally. So that's the execution part. But now like imagine I tried to find different APIs and they all give me some kind of execution outcome. Which one should I use as my ground truth here? So this is what we call a filtering process.
38:13 The process is also very kind of straightforward. So we are going to make this as a learning problem as I mentioned. First I'm going to share you the loss function here. This is a very classic a cross entropy loss on the tokens from position I and onwards. So basically J starts from I. So everything that I'm going to generate from the current token till the end. One minor thing different from the classic cross entropy is that we also add this W like weight here. this basically decays with distance. So, tokens right after the call actually count a more, but tokens like really for further away actually does not count a lot.
39:02 So, then what does model does to kind of filter this is literally a comparison. Okay. So, we have LI plus. This is that we are going to give not only the current prefix, but also the tool call and the execution output. So, this is a complete package. This is our LI plus. LI minus is like two baselines. What if I just do not do API call? What's the loss function? What if I did API call, but I do not have output?
39:32 What's the What's the loss function? And I take whichever is the better and then compare it with my like finished tool call loss. If this is bigger than a threshold, then I'm going to keep this specific API call. So, this is actually what's happening here. let's look at a one example. so, the loss function that I mentioned to you that weighted decay cross entropy. So, let's see we use it to calculate my example of Pittsburgh is known by what? And then the prefix is Pittsburgh is known by. And then let's see we got a Steel City on this kind of like QA API call.
40:17 So, the condition is that I'm not going to do any API call. I will just have this prefix and then you get your loss 2.5 here. If I get in the QA call, but I didn't give them the answer. This is what I like one of the stronger baseline there. You can see that my loss is only 2.1. If I executed this QA API call and also give them the answer. And then this give me a loss of 0.8.
40:47 Loss in general is lower the better. So, here if I look at the usefulness of this API call for the current situation, it's actually 1.3. It seems pretty good. So, I'm going to keep QA call as a choice here. Of course, you can do this for many other API calls and select whichever give you the smallest loss. >> My question is you're being very clear with why. So, it's not just a textual comparison.
41:19 You are actually executing the call, getting the response. So, if it was caching the answer to these questions cuz you don't have any control over what the server's doing. I guess my question is how robust will this be cuz you could easily just get cached answers. >> Yes. So, a few things. First, you are right that here everything need to be executed by a external system, right? And then when they execute that, you fetch back the results and then you see how good is this fetched back result.
41:54 This is what's happening here. And it's totally plausible that I called this QA server, but the QA server was down. So, I get a nothing back. So, you just use whatever has been produced here. Or you you you you give it, but then because your format is wrong or whatever, you didn't get anything. You get a error or the number is wrong. And then that's also the cost that you would have to pay here. So, this is why this learning-based approach may not be the best and then we'll see the evolution of how things go.
42:26 Okay. I hope that part is clear like in terms of how they determine where to insert API call and then how to determine which API call to use. And overall like if you just look at a more examples, so I give you the usefulness score that is the loss comparison. So, you can see that the first the one the WIO be open on Friday, March 10th. So actually they can figure out this with a right API call. And then the useful this score of this one versus the last one is not very useful. So you can see the score is pretty low. So this is actually how you are going to determine whether a particular API call is useful in a position or not.
43:12 So two formers actually work pretty well when it was first introduced. So on my screen you see five different type of task, fact checking, question answering, math, temporal, multilingual. you can see that models are doing extremely well, especially like two former is probably like many times smaller than GPT-3. you can see that two former is actually trained on GPT-J here. So it's much smaller than GPT-3. you can see that at least the two former results on math is just like so much better on temporal information, even on factor completion, on multilinguality. So it's just like a much better if the model actually know how to use tools, and they can actually outperform a very big model.
44:03 >> Okay, so I think this is this actually like how model agents system actually work? That basically inside of an output token stream the model decides to switch to a tool or is it more like like after one agent turn the agent decides okay I want to use these tools and then another agent turn it does the output. do you understand my question? >> Yes. So here the fashion again I'm going to show you a evolution of how the agent the tool use evolve. At what I'm covering right now is actually like when you generate each token, one of the token may trigger the tool use, and then you put it that as part of your generation. So this is the current fashion. Later when we go to the very agentic systems today it's more like agents actually decided that okay I need to use a tool for this particular question. And then they retrieve one tool, and maybe it didn't work, and then they go to another tool, etc. So you get a trace of how agent use tools.
44:59 Okay, let's continue. Since this is a very classic work and the people are already eager to ask a modern questions. so even for a model like a GPT-2 size with and without two formers, the change is actually a lot. So the blue curve is actually when we have two former and then the right is without a two former. And the dash line here GPT-3. So you can see that this is actually a very important design.
45:29 going back to some of the earlier question that is oh, this is a fashion where the model is generating next token and then suddenly we let them to do this kind of tool use. And apparently since this has a lot of connection to words, tokens, there was a lot of work talking about what if we just view tools as words. And then there's this study called a token GPT. So you basically view each tool as a token. And then you can actually learn a embedding for it.
46:02 So whenever in the generation process a token, not token, a token is triggered, large language model will complement the arguments and then for the tool to execute. So now we actually really use it in token-based fashion. So I'll give you a quick example. We'll be very fast here. So imagine you are solving a kind of like a math problem here and then at this point the model needs to generate the answer for this math question. And then they decided that I need to use GCD function here. And then they they trigger that.
46:38 They use the embedding like that embedding get triggered and then they do this kind of calculation. So as a result you can see that there is this reasoning mode. There is this tool mode. There is this kind of plugging back mode. We have talked about like wording bearings briefly. so wording bearing the key idea is that okay, how can I get a dense representation for each word? So here, we also want to just view token like token as a regular token.
47:11 this is a process of how it got triggered. First, you predict a token here. And then once the model see that I need to use the tool, they stop the generation and they're going to switch to something else. they look at the available tools and to decide which tool I want to use. What is the arguments, the parameter I need to give to my tool? And then they will give this tool to a external model like in-context learning or others. And then they get the results, ask you that, and then bring the results back. And then it goes back to the original generation. So you can view this as whenever they they realize they need to do a tool call, they pause, they do the tool calculation. Once they get the results, then they continue. So this is the process of Tool Chain GPT.
48:02 very briefly, they do a lot of synthetic data to kind of construct this training coppers and then they can actually train the specific token embeddings. this kind of Tool Chain GPT also works well. one thing I do want you to notice is that you see here I have one hop versus multi-hop here. Multi-hop means that the agent asked to use a sequence of tools.
48:35 It's not like enough if they only use one. And then if you look at the numbers like one hop is 0.7, 0.5, it's pretty high. But then if it's multi-hops, this is literally like 0.03, 0.09, it's pretty bad. So this is actually a big challenge earlier for tool use. That is, if the agent needs to use a sequence of tools, this is actually much harder. Okay, I'll stop here. You all just finished the most math heavy part of this class. Any questions?
49:09 Okay. Yes. >> Why are multi-cops why is multi-cops like really bad? Like is it is it on the two-sided problem or is it the positional embedding problem? >> Yeah, I think there are many reasons of why multi-cop is more challenging. First is that if you just view this as like each step is a degree of freedom. Many steps, you just have so many choices there, right? Like the combination of possible sequences is just huge. This is the one. Second is that I think in a lot of the like what I was talking about how they train the model, they do have some kind of ground truth data. If you look at a ground truth data, a lot of them is actually just like a single tool call. The reality is that as AI gets better and better, we expect them to do a lot. And that a kind of sequence of tool use is just not a very well like represented in the natural world. So that's another thing. But I think the major reason is just like the search space is huge.
50:13 Yes. >> I wonder every day there are new new tools out there being created. How does how does it even know what are the new tools out there and how do you explain and which ones are >> Okay, great question. We will talk about that in 10 minutes. Okay. so I will continue. So what I shared here is really just like how two cop get introduced to us, like two years ago. And then a lot of things have changed.
50:40 First is that you probably noticed that in the two-former, they only use five tools. But we actually have a lot of tools. How can we actually scale this to many many tools here? the first the work that I want to share with you is actually called a tool LM. They actually enable AI to use over 16,000 APIs. the way how they do it is actually very straightforward. You can also view this as a modern fashion of self self-supervised or like like a synthetic data construction. So, what it does is that it first just crawl a lot of APIs because those are available online.
51:23 And then they come to ChatGPT. They ask ChatGPT like, "Okay, here is a tool use. Pretend that you are a user asking for something that would trigger this tool call. Give me the input and output." So, they actually work with AI with our arms to generate this kind of synthetic data. Okay, once they generated this kind of input request and then which kind of tools to potentially use, they can then ask ChatGPT again like, "Okay, now pretend that you have this question and you may want to use this kind of tools.
51:57 Just try to solve it." Then the ChatGPT is going to like use this tool solve it and if it didn't like succeed, try different tool etc. So, there is actually a tree series of how AI solve this kind of complicated user query. Okay, so that's actually the data data set that called a tool bench. This is literally a self or machine constructed data sets where we use AI to simulate user queries. We use AI to solve how to do it. Okay, now you have a ChatGPT h solution trees of how to do tool use.
52:38 Well, then we can do the simulation. We take a llama, which is a earlier model from Meta, and then we'll just do SFT on this like ChatGPT reasoning tree series. We can then have a pretty capable tool llama that knows how to do tool call. Okay, so this is a key idea of two LLMs. And then at inference stage, what the model does is that they get a instruction from the user, they look for different APIs, then they give it a tool the tool llama and the tool llama will produce the final answer. So, this is actually how it works. This is a constructed data set to give them over 16,000 APIs.
53:24 Yeah. >> >> Okay, so this kind of like distillation.
54:09 So, first I want to make a claim that this is not the classic like Jeff Hinton H knowledge distillation. This is a new wave of distillation where we have a very capable model and we know if we know their reasoning traces or if we know how they solve the problems, we can actually do SFT to enable a much smaller model to have similar capabilities here. if you pay attention to news, you actually see a lot of discussions about distillation these days between different the frontier model providers.
54:43 Going back here, so I know this may seem like, oh wow, this is even like is this even possible? Like the data seems not even not a very high quality or reliable or whatever. The reality is that you can actually do this kind of synthetic data generation where you you pretend a user is going to ask some question that it involve this set of tools. So, you then try GPT can help you write a search a question because this is a very clear task. Not the tool ask them for their reasoning choices, but ask them to do a specific task, right? Once they did that, and then you frame it as, "Okay, now this is a question. You may could use this set of tools. How would you solve it?"
55:26 Again, the how is actually a process. They are going to describe like, "Okay, I'm going to do this and do that." still we are not attaching the internal reasoning choices. So, slightly different here. and then once you have this kind of synthetic data, your training set, you can train a much smaller model on top of that. >> Exactly.
55:57 >> Yeah. So, you basically use them to do very specific user instruct task. And then maybe you need a multiple steps to construct a training set. Okay, I'm going to continue. so, this is about the two arms. And then one thing actually occurred in this process. So, this is again even at the GPT-4 like more recent at Cloud stage. if you want the model to help you with speech-to-text, they need to call certain type of models to do this, right? And then on the left side you can see that here a particular model they call it from TorchHub is actually ASR and the source equals local. Those two are actually wrong. Like it does not exist. So, the model actually hallucinate certain type of parameters in the process.
56:44 In the middle, Cloud was also trying to do the same task. And then they said that, "Okay, you are going to use translation here." The user wanted speech-to-text. So, they actually called a translation model here, which is a wrong library. So, the model actually used a wrong API here. especially when they are so similar to each other, this is where issues may occur. And then what do you really want is actually speech to text. Go to torch hub and load this particular transcribable model here. So, now like how could we really teach models to use a lot of those tools and especially have a way to quantify hallucination in the process. So far, we haven't talked anything about how to quantify hallucination in two years. It's only like correct or incorrect, but is there a way for us to even do this?
57:36 So, the Gorilla is actually a system from Berkeley where they are trying to connect LLMs with a lot of APIs. By a lot, I mean 1.6 thousand here which is actually 10 times smaller than 2 LLM. So, what they do is very similar to the construction I just mentioned. You are going to scan a lot of the APIs. So, here they actually focus on a lot of the modern AI tool use, hugging face, the model hub, PyTorch hub, the TensorFlow hub, and then they get a lot of API calls in JSON format. And then they use GPT-4 to generate a synthetic instruction data.
58:19 So, here's a query, here's what kind of tool you should use, etc. On top of this synthetic data, just like how I mentioned with 2 LLMs, they are going to train a model. But the training here may not always be trivial because if you only train the model with correct docs in the prompt, then the model only learns to copy whatever it's given and a wrong doc actually will lead it to the wrong call. So, the way how they deal with it is to do this kind of retrieval aware training where they introduce both correct and incorrect retrieval during the instruction fine-tuning process with the goal that this kind of setup will teach the model to first to recognize what is correct, what is incorrect retrieval, and then do a lot of the generation on top of that.
59:12 So, at inference stage, like here, let's say you can do a zero-shot inference. If the user says that I want to see some cats dancing. the model is going to look at all the available API database and decided that, "Okay, I can actually call stable diffusion pipeline here." And then you call stable diffusion by giving them this query. And then you get a specific result. And then Gorilla actually return that. And then from a user side, what do you see is only this execution results. Everything that I just described are just like in the middle.
59:48 Today, that in the middle just means that a lot of token, a lot of like cost produced. so, how would you evaluate an API call? If you are going to do string matching, it's going to be very strict, very hard because those two call, like here, it's actually very detailed, like 1.1 version is very different than 10.1, etc. so, what this work produced is that we can actually pass the generated call into a syntax tree.
60:23 For many of us, when we learn programming earlier days or like computer computer graph, you can actually turn many of them into a graph representation here. So, on the left, this is actually a syntax tree of what do we have here for this particular API call. And then we are going to check whether this is a subtree of the tree that we are going to build from a real API database. So, if it match, then at least it means that this is a valid valid API call.
60:56 so, you will also get us some other numbers. First, if the tree does the subtree does not match, then you know that the model probably hallucinated an API that it does not exist in my database, right? So, it give you a very intuitive idea. Or maybe it a but it give you a wrong node or something like that, then that's a error. You know that this is a error, not a hallucination. to show you the results, first, definitely they are better, than a lot of the baselines they compare.
61:30 What I want you to see is actually this figure. So, the zero shot here, you can see that if we quantify hallucination of API call versus accuracy on a like a parental analysis figure here, Gorilla, because of this like AST-based like setup, they're actually having a much lower, hallucination and a much higher accuracy compared to many of the other models. So, this is actually one way to teach models to use certain type of tools.
62:03 Okay, so now I I actually shared a lot. Those are actually four classic learning frameworks to think about how to get let the models use tools. And then, again, like take a many steps back. this is again like a a learning approach, which means that we want models to know what a kind of API to use, where to use, what type of things to use, and they need to know how to fetch back the results. So, this is actually a very different thinking.
62:34 but the issue here is that if, let's say, as some of the question asked earlier, like today, different API and the tools come out every day. whenever someone introduce a new API, if you train the model before, now you need to retrain your model. This is actually very costly. Or, you need to retrain your word embedding, which is not going to be very flexible as well. So, what if what if instead of training the model, we are going to change how those tools are described, so that any model can actually use them at runtime?
63:11 That is, I'm not going to get my model to know what a kind of things to use as a intrinsic capability. I'm going to make sure those tools are specified in some kind of like universal format so that as long as I can have a conversation with the tool, I know what kind of tool to use, what are the functions they have, etc. So, this is what I I'm going to share next. That is it's actually a not a from a learning perspective, but more from a industry or like a easy-to-use perspective. This is what we call standard interface.
63:48 so, just to kind of recap what I have said or like why we need a standard interface. what I have covered so far is we are changing the model to fit the tools. And then basically every new API means that we need to do another training. But now what I want to share is that in practice, what it works better is that we are going to change how the tools are described so that any model know how to use that.
64:17 So, this is going to be a mindset shift. Also means that no more math for the next 15 minutes. Okay, so I'm going to share you three generations that we have. so, this is also where a lot of debates about tool use happen. Like there was also even a lot of debate on whether we should redesign the internet so that the internet is going to be agent-friendly because a lot of the APIs we have were human-facing.
64:44 Okay, so the very last thing I said is this is actually a human-facing tool called the REST API. a lot of details that you can learn about that. I will have examples next page. But this is literally you give like the model or like something string and then they are going to read it or ask you that. And then this is this this is what's happening. And then you can see that the process is also very human-readable. So, it's like get and then weather CT equals SF here.
65:21 So, function call is slightly different. It's designed for developers or like designed for the model. So, each tool is going to get a JSON format and then the model every time just emits the name and arguments. And you can also have a harness around it to validate whether the call is is specified correctly. and here again, if the tool change, the function call needs to change as well. Okay. So, that's probably also not a very like very easy to use. So, what if we just have a lot of agents now and they all want to use different type of tools.
66:01 So, this is actually from an ecosystem facing perspective of MSCP. model context protocol is designed for the ecosystem. We will have a server that expose its tools. I'll show you the tool list discovery. Then others or any client can actually discover that at the run time with like tool list or or invoke them with like tool use to call. And then the server is only need to be written once and then reused across many different host.
66:33 So, I guess like plain language summary here is that rest those are more like English docs like written for humans. Function calls are written in a JSON schema. So, that the models can read. And MSCP, they are written in a protocol format that every model, every application can reuse. So, let's just look at some examples to make it a very clear. So, this is a very classic REST API. A lot of them is actually over like HTTP request. So, here if I want to know the weather at a Stanford today, you just use a get and then give them this kind of URL query. And then you get a code 200 means that success. And then it give you the specific temperature.
67:22 This is actually what it will look like. there are four different actions or even more. I think this is the whole vocabulary. Like get, post, put, delete. So, those are very limited actions you can do. So, the limitation for a lot of the REST APIs that the model must know the URL parameter names and response shape. So, you need to get all of those information there. the second generation, as I was talking about, is like a function call.
67:54 That is, what if we actually could get those tools in this kind of very structured JSON schema? This is actually a concept, I think OpenAI introduced in around 2023 in the summer. And then after that, many companies follow this tradition. So, the key idea here is that we give tools to the model as some kind of JSON schema. Since like JSON is just much easier to work with. and then the model is going to answer with a tool name and type of typed arguments. instead of some kind of free text here. And then the code is going to run the call and then return the results. So, here for the same query, instead of a URL, we actually give them a schema. And then the model is going to give us some kind of output also in JSON format. This is just like a much easier.
68:44 Very earlier days, actually this kind of JSON schema is a great way to do prompt engineering because if you just ask the question, it's really hard to get it correct. But if you let them to return the output in a JSON format, you get a a little bit more luck with it. so, again, the limitation here is that since this is a tool call, there is a tool schema, etc. if I have a new tool, then the API or this kind of JSON schema needs to be rewritten, which is not a very scalable.
69:18 this is why like the third generation we are talking about MCP. so, it's one protocol between any AI applications and any tools or data source. So on the left you can see that we can have chat, cloud desktop, IDE, cloud code, or other AI applications here. And then MCP sits in the middle and then on the right those are different data sources. This could have be data on the file systems or even your Slack or developer tools such as Git here.
69:51 And then this is just like a very interesting setup because with function call or function calling we need to do this kind of JSON schemas. Here it seems that we just need to figure out how to do this standardized protocol. so if we go a little bit deeper here and there are a few terms that you will encounter. So MCP host is the is the app that you tap into such as like cloud desktop. And then a server is like a tool provider.
70:25 so your local files, your database, remote service. And then client is actually the connection object the host actually creates for each server. So you will have one per server. It would does the handshaking, deal with the tool list protocols, return results, etc. So you can see that here we actually have four clients here. Means that there are four open connections we have, not four users. It's like four connections.
70:55 you also see that like server A serves file system, server B deal with database, and both client three and the client four actually link to this specific server C here. So the host never learns anything about a specific file or database format. It only need to know how to speak MCP language and then they talk to it with this kind of MCP protocol. So you can view it as like a one layer abstraction. So, let's see you just established a connection between the server and the client. And then, how do I know which tools to use?
71:37 In the past, we need to get you to train a model to figure out which one gave the lowest loss. Here, you are just going to ask, "What are the tools do you have?" And this is like tools/list. The server is going to return you a list. And then, each tool will have a name, description, a JSON input schema. And then, your job is to decide, "Okay, based on the description, is this the tool I want to use or not?"
72:05 There is another like tool execution that we keep talking about. And this is like the tool/call in MCP. Basically, it tells the model that, "Now I know which tool to use. Run this tool with those arguments." You need to first pick a tool using the name. And the arguments basically just like fulfill all the input schema. And then, you get a certain kind of results. So, this is actually the process. One matter of thing, if you are just reading this or hearing this for the first time, you probably notice that, "Wow, this format is like insane. Like, am I going to write this kind of like code in my in my code?" The answer is no. We are not going to write it. As I was talking about, like MCP being a protocol, we want the models to actually generate those kind of detailed like schema stuff like that. We are only in charge of some kind of a high-level specification.
73:07 So, let's let's look at a real example. So, let's see our course actually has a like learning management system. And there's an API. You can have like assignments submissions or deal with grade. So, in the MCP fashion, like there is this like a package called a fast MCP. So, you can basically just tell them like, " this is the specific API I want to visit. And then what is the specific like specification I want to give?" And then you establish this. Then the model is going to let's see if some some of your agent want to access our course LMS, and then they are going to see what are the tools do you support? And then our course will tell you like we support to give you the list of assignments, get the submissions, or even post the grade.
74:02 Okay. This seems fine, right? But then think about any of you can build the agents to access our course website. And then post the grade is probably not a something that everyone will have access to. So, in addition to also enable the connection, we could also control who has access to what kind of tools. So, very simple again, let's using this as an example. You can see that the course MCP server actually sits between your agent and then our kind of like rest API here.
74:37 And then there is going to be an inbound kind of check whether you have access for a specific thing. But again, there is also outbound here. one interesting thing is that agent never sees my like LMS token here. And likewise, like API you never see the agent token. So, everything is like kind of separate. So, you are only allowed to use whatever tools you are allowed to do. Anything beyond that, we will get it with like access or off here.
75:08 So, this is actually what do we have in terms of standard interface. it may look like a lot of content, but it's really just like protocol change. And people have been using them very often these days. And so, it's a very practical thing that you can play around. So, in the next 3 minutes, I'm going to quickly mention a few tool use data sets just as an information for you to know where to look at. this is actually the live leaderboard from Berkeley called the Berkeley function call leaderboard. So, if you care about the function call, this is actually a great resource to check out.
75:45 They benchmark different models in terms of the data set. so, the evaluation here, we already briefly talked about that. So, you can do this abstract syntax tree matching to quantify hallucination. You can also look at the execution to see whether you get it right or or wrong. you can also do like a lot of like state-based evaluation for multi-step or multi-turn function call here. Their data set is created by looking at a lot of real-world use case established from different taxonomies.
76:18 they also try to insert some situations where you can also check for a specific query, if the answer is no, there's no not available tool, or no, I cannot deal with this, model knows how to fact like factor it back. Not only just to give you a like fixed query or API. if you are interested in checking out the data set, I think like this is literally what a function call look multiplication, parallel function, or even like a relevance detection here. So, a wide range of function call situations.
76:55 I already talked about the tool bench where they use this kind of synthetic data construction to come up with how ChatGPT solve a particular user query, and then you can use it to quantify whether your systems can use tools correctly or not. tool bench is actually another widely used approach or data set. It basically introduced the role of customer. So, now you have a agent, and then the customer is interacting with the agent. the customer may want to cancel their flight or may want to change their flight, and then the agent needs to know which correct tool call to use to make the action happen. So, it's actually this kind of tool agent and user interaction data set.
77:41 last thing that I want to mention here is if you think about evaluation. For a lot of retrieval we talked about the last class. We use pass at a K that is if you tried it a K times, if you succeed once, then that's good. However, when it comes to this kind of multi-turn interactions, we actually want reliability and the consistency. The user is not going to try your systems eight times and then as long as one time work, they are happy. As long as the user tried it, it didn't work, user is going to be very angry, right?
78:14 So, this is why we need a new metric called a pass hat at K here. It literally means that if I tried it at K times out of N, I want all my K times to succeed. So, this is actually a much stronger metric. Top Bench the first version include the retail and the line. You can see the numbers here. what I can share with you especially in the bottom is that pass hat at K and pass at K has a very different trend.
78:44 So, this is why we need to be very careful in terms of what the metrics we are going to use. Top Bench also has evolved a lot. The one that I've mentioned is from 2024. You can see that over the last the two years, now we just have Top Square Bench, Top Knowledge, Top Voice, a lot of those. So, with that, that's all what I want to cover today and thank you. See you next class.