The Rise of System 1 Models | First Pass Ep. 11 with Sam Crowder (LangChain)

0:00 I think normal LM architectures we're going to see get like better, faster, and cheaper over time. So that will that will simply help. I think system one models also will play a big role though. Welcome back to First Pass. Today we're joined by Sam, head of product at Lang Chain. Sam, thanks for joining. >> Good to be here. Thank you guys. >> Everyone's talking about Jev OpenAI just launched decisions API system one models are are really taking off. for listeners who haven't really been following, could you explain what a system one model is, the emerging system one versus system 2 architecture and and why does it matter?

0:39 >> Yeah, system one models are basically a simplified version of the large language models we've gotten really used to using in various ways over the past couple years. so whereas you could say system two or large language models are like open English open text as input open text as output system models system one models instead take in like state or context and various questions as a slightly more structured input and then give a very structured output as a response.

1:06 >> And staying on that topic, what kind of decisions should move out of a traditional LLM call and move into sort of the faster system one decision layer? Yeah. So basically what you have with system one models is like a really really fast and really cheap way to get decisions or probabilities. And so while it is very new, there's all kinds of potential applications. because if you think a lot of or really all of traditional software is deterministic, like still a lot of the decisions that we want to make in the digital world are more deterministic in nature. and so think like ticket routing or even model selection for large language models.

1:42 >> Same, how would that work? using something like Jev like let's say you want to do ticket routing you like could you just walk through like what would be like a before and after like h how how would you use a model from anthropic or openai versus a system one model like Jev >> yeah so how you typically use a model from anthropic openi is is you would basically take whatever state defines that let's say ticket and turn that into some kind of pros often like paragraphs and you'd pass that entire block of text to the completions or the responses API that they host and it would give you back basically text in response. you could also do like ticket routing via structured output with these traditional large language models. But the way that actually works is the system two models auto reggressively generate text and then sort of cajol that text into a more structured decision. whereas the system one models more natively take in that state. They take it in a slightly more structured fashion. So it's not just open text but it's basically a context variable which could be a string could be could be JSON and then it's also a set of one or more questions and and the system one model architecturally is baked to just directly give that decision or probability as opposed to generating text first and turning into structured output.

3:00 So going back to that ticket routing use case, what would you use the system one model to do and then what would the system two model do as part of the application? Yeah. So the system one model you would basically take like a linear ticket and you would turn that linear ticket into a state variable usually still just a string. So like a block of text and then you'd ask a question and the question might be like does ticket d-123 belong on team evals or team AI observability and the system one model would give one answer to every one question that you pass it and so just choose one of those and give sort of like a probability of its own confidence in that selection. whereas using a more traditional large language model, you would you would structure the question actually very similarly where you might turn the linear ticket into a block of text send it to their API and then it would it would under the hood auto reagively generate that text as like sort of a a single sentence or statement answer and then if you did configure structured output over that API then it would turn that into a very similar decision but it would do so much slower and much more expensively.

4:13 And so is that that the advantage? Is it you know you get a better system? Is is that from like a cost perspective, from a better performance perspective or or latency or or all of the above? >> Yeah, it's really it's latency and it's cost from an API design perspective like system one models are strictly more limiting like with traditional LLMs that we're more used to using like you can ask all the same kinds of questions and more because you can also have it generate like marketing blog. but basically because system one models have been much more opinionated in the types of things they should be used for, that comes with architectural advantages.

4:46 >> Got it. And what what are the trade-offs? Like what are what are some of the downsides of of this approach? >> Yeah, I think it's just sort of like we're saying like it's pretty use case dependent. Like if I think about canonical like early post chat GBT like use case of LLMs like you know J Jasper was a company a lot of people were talking about using at the time and like you would use that as a person in marketing to generate marketing copy system one models are much more limited and basically can be thought of as like a way to make structured decisions or even as as a thing you should you should use in tandem with more traditional LLMs like I mentioned model routing as a use case. one thing we worked on last week and published here at Langchain is we have our open- source agent harness which is called deep agents. we built basically a way for for developers to do model routing using system one models as part of the agent harness. So basically if a question comes in, you would first very quickly and inexpensively pass that question to the system one model and the decision you might be making is should I route this to to opus or sonnet. and then it would output that decision and the harness itself would then route to the LM to do the actual generation of text.

5:58 >> Can you can you kind of compare and contrast that to the decisions API that OpenAI released yesterday at Devday? I mean it it seems it seems similar at at first pass, but I' I'd love I you know I'd love maybe a more experienced perspective on it. >> Yeah. Yeah. Super similar. I think I think it seems both type safe and openai and I'm sure anthropic will continue to learn from each other and and push each other in ways that are good for the industry. Jev as it was first released is is text only. and the openi decisions API is multimodal so does text plus image. I'm I'm sure all the providers in this space will you know soon have all kinds of modalities but that's really the core difference right now >> in so in the in something like a classifier or decisions where the output is is hey who do I route this ticket to where is the enduser kind of like defining success or you know how does the model know hey it should be routed to the observability team versus the eval team like what type of configuration is needed for some of these models.

7:08 >> Yeah, I think it similarly to working with traditional LLMs like it comes down to how you prompt the model. So like you you might prompt it with like two shot examples saying like here's an example ticket and this one belongs to the eval team or this next one belongs to the AI observability team. so that's usually where where developers start very similarly with system one but also system two models. where it gets more powerful over time is this flow that we in really more in the eval industry refer to as like aligning an evaluator which is like I I sort of can write my initial prompt and it maybe makes a bunch of decisions and then I would go through those decisions and correct them one by one and then use that to tune the prompt so over time the the prompt you're giving to any any kind of these models is more aligned to your preference as a domain expert. Is it fair to say that this is, and I think you kind of alluded to this earlier, like a good step towards making these systems feel more deterministic?

8:04 >> Yeah, I'm I'm honestly waiting to see and excited to see how it unfolds. Like I I mentioned the the way we're using system one models in our open source harness to do like model routing. That's basically using Jev as like an assist to an assist to like a fully non-deterministic just looping agent. Another pattern we're exploring is sort of a a hybrid of like traditional software which is like fully deterministic and agents which are fully nondeterministic which is we have an open source package at Langchain called Langraph which is durable execution useful in a lot of agentic but also like LM workflow type applications.

8:43 and and when you think about durable execution like a lot of the decisions you have to make along the way is like should I route the next step of this process to node A or node B which you could do via like deterministic code or you could do via like a very expensive LLM call or now you can do via system one models. So that's how this sort of like hybrid architecture emerges where it's a mix between determinism and non-determinism. and we'll see how popular that becomes over time.

9:10 >> How do you think things look in the future? Is it is it going to be a bunch of chain together system one models that kind of create workflows and agents or or are these system one models more of the kind of like the top of funnel as part of a broader workflow and and and kind of general purpose LLMs like still like eat a lot? I mean maybe the question of the question behind the question is like hey what does this mean for big frontier models? but yeah would would love your perspective.

9:38 I'm I'm I'm closer, if I had to predict, to the second version of what you pitched there. I don't think I don't think most software is going back to determinism or back to like semi-determinism. I think like non-deterministic software is here to stay. I think normal LM architectures we're going to see get like better, faster, and cheaper over time. So that will that will simply help. I think system one models also will play a big role though in in largely routing questions in terms of like what what is the best LM to handle this question in the moment >> and and for developers looking to get started with system one models today where where should where should they start?

10:14 >> Yeah. we since the the founding of this company almost four years ago have been in the business of making it easier for developers to build useful things with AI and doing so in a in a fashion that is fully pluggable. Use whatever model you want. so we have really three main open source packages which are lang lang chain lang graph and deep agents. lang graph is sort of the durable execution that I described. deep agents is is like a fully non-deterministic agentic harness and then lang chain is sort of a simpler thing to get started with out of the box. we're building system one or or Jev type models into all of these frameworks and I think that's a great place to get started.

10:53 >> Awesome. Well thanks for coming on Sam. >> Thank you guys. It's good to good to see both of you.