Building a Harness with Jev

0:00 Hey folks, my name is Sydney. I'm the product manager for the open source team at LangChain. And today I'm super excited to share with you how you can build a harness with Jev, this new model that just came out from TypeSafe AI. So before we dive into what Jev is, why it's so exciting, I want to do a quick recap of the core agent loop that we're all used to. So a request comes in. We send that to an LLM.

0:24 And it's able to take action by using tools. So it will call a tool, get some results back, process those, and then it can continue in this tool calling loop, or it can decide that it's done with a given task and return a result. This loop is super powerful, but LLMs are a little bit unpredictable, right? They get text in and then produce text out. And so when LLMs first came out, we decided that we needed ways to impose a little bit more structure on them so that they could work well in code-driven applications that inherently depend on structured types, not just the raw text.

1:02 So enter tool calling and structured outputs. Again, tool calling, a model can send a structured request. In this case, a get order request with a given ID, and then the tool returns a result with structured data. The other primitive we added, again, is structured outputs. The idea here is that you can bind an output type to your model, and so it produces a final result that adheres to a given JSON schema rather than just raw text.

1:29 And so these two primitives made it a lot easier to take advantage of this really powerful loop in code-based applications. Okay, so now to the exciting piece, what is Jev? There's so much buzz right now on Twitter about this new model, why is everyone talking about it? So Jev is what we call a System 1 model from this new startup that just emerged called TypeSafe AI. So according to their docs, System 1 models are a class of AI models built to make fast structured decisions that software can use directly.

2:01 A System 1 model evaluates a state and questions and returns typed answers and probabilities. You can see an example of state and questions and then typed answers here on the slide. The interesting thing here is that Jev actually doesn't take text in and then generate text out like the traditional LLMs that we're used to. Part of the reason why Jev is so popular right now is it can be 20 to 200 times faster and 40 to 400 times cheaper on classification style tasks compared to an LLM.

2:35 So Jev is not a drop-in substitute for an LLM, but it can do specialized tasks that we currently ask LLMs to do much faster and much cheaper. In case you're curious where this System 1 phrase is coming from, it actually comes from psychologist Daniel Kahneman's book, Thinking Fast and Slow. And in this book, he described two ways that minds think. Fast and cheap, System 1 intuition, versus slow and expensive, System 2 reasoning. And so TypeSafe AI borrowed this sentiment for its new model called Jev, calling it a System 1 model.

3:10 That's fast and cheap, returning almost instant typed decisions instead of generating text, which is in deliberate contrast to standard LLMs, which behave more like System 2, slower and pricier, but adhering to more of this step-by-step reasoning over open-ended problems. All right, I'm going to show a quick demo here on how an LLM answers a question versus how Jev answers. So the question is, is there PII in this text? And so an LLM produces this text result, and then also structured output takes about five seconds.

3:48 Jev almost immediately, chance that there's PII is 98%. And so you can see kind of from this demo for simple classification style tasks, Jev is super useful. And there are a lot of classification or kind of decision style tasks that we currently ask agents to do with LLMs that could be more optimal in terms of speed and cost. All right, so let's talk a little bit more about the types of questions that Jev can help answer.

4:15 There's three types of questions, and we'll look at them through the lens of this example. So the state says, "Hi, I've been trying to connect my Stripe account for three days, and it keeps failing. I'm losing sales. Please help ASAP." So question type one is a choice. Multiple choice, pick one option. So for example, which team should handle this? The score shows 0.84 for billing with a confidence of 0.596. The second question type is a score. So somewhere on a scale, you know, how frustrated the customer appears from calm to frustrated to very angry. And the score here is 1.035. So the customer seems frustrated but not very angry. And then the third question type is a boolean. So this is a boolean type answer, a yes or no. So the message conveys urgency or time sensitivity returns 0.999. This is on a scale of 0 to 1. So certainly a high urgency probability for this case. One other thing to note, you can actually send a single state with many questions that Jev answers in parallel. So Jev is actually very efficient with many questions at a time. This is again in contrast to LLMs, which do more sequential reasoning rather than parallel processing.

5:35 All right, so how can you use Jev in LangChain? We just released a LangChain TypeSafe integration. So from LangChain TypeSafe, you can import this TypeSafe classifier. First, you'll need to acquire a TypeSafe API key, and then you can use this classifier to invoke Jev with a state and questions. And then the response has answers to each question that you send it. Let's walk through just a couple of use cases that are super great for Jev. The first is model routing. So if you have an agent that does both simple and complex work, you probably don't want to use the most powerful model for everything that the agent does.

6:17 An example of this that we've been exploring at LangChain recently is that we actually want our internal coding agents to switch between fast and cheap models and more expensive and powerful models depending on the complexity of a given question or coding task. And so Jev is great for this model routing step. You can assess a given prompt against given criteria to help decide whether or not you should use a fast or more powerful model. And Jev decides almost instantaneously.

6:47 Another interesting case is what we call auto mode. So from LangChain, we already offer this as a pre-built auto mode middleware. And the idea here is that you can ask Jev to analyze whether or not given tool calls are risky, and then you can block those at runtime. Anecdotally, I actually turned off auto mode in my coding agent recently, because the classification step of whether a given tool call was risky was too slow for my coding agent to feel productive.

7:14 But it's back on now that Jev can make these decisions so quickly. So a quick example here is like if a tool call was issued that would delete a database or delete important files, Jev would certainly classify that as risky and then block that tool call. All right, the third interesting case here, maybe the one I'm most excited about is using Jev as a judge for online evals. So the idea with this is that you often want some sort of model to help you identify whether or not your agent has done a good job with a given task.

7:44 But when you're running evals at scale, it's not super reasonable to have human oversight over every single trace, but you often need more than just kind of code style evaluators to decide whether or not your agent is doing a good job. And Jev is really great at this. So you have your input for a given eval, so the initial question and then the answer from the agent. And then you also have a rubric that you provide.

8:09 So grading criteria, like, is this answer correct? Does it match the reference? Is it grounded? Was there a cited source? And then Jev can score the given answer along these different rubric criteria. This is kind of an evolution of LLM-as-a-judge online evals, which are currently certainly very useful, but also very expensive. We just wrote a blog on how to use Jev as a judge, and it has some really interesting results about Jev being, surprise surprise, much cheaper and much faster, but also interestingly, much more reliable and consistent across evals than the LLM alternative.

8:47 All right, so I'll leave you with this. It's time to go build something cool with Jev. You can get started with uv pip install langchain-typesafe. You can read our accompanying blog post at the link here. Easily get an API key from typesafe.ai, tag us at @LangChainAI on X. We'd love to see what you're working on. Thanks, folks.