Johann Schleier Smith - Systems Foundations for Agentic AI

0:02 Hi everyone, I'm Johan Schlersmith. I'm at Temporal Technologies and today I'm going to be talking a bit about our perspective on the agentic AI landscape and introducing Temporal as well. So just by show of hands, who here is familiar with temporal? Maybe has heard of it? Okay, we've got a few. How many people have actually used temporal? All right, well I'll tell you this is going to be a highle overview. If you stay past this session, we're also going to be doing a workshop where you can get hands-on and get into the code. So, let's just get into it and let's talk about systems. So, when we talk about systems, we're talking about creating some sort of functionality by putting together a whole bunch of components.

0:44 And in the case of AI, there are a lot of these components. And so, we need to get that functionality. But in addition to the core basic functionality, there are these sort of crosscutting concerns, right? things like reliability, which Temporal has a really a deep experience in efficiency, security, evolvability, that includes the scalability. it includes the ability to do upgrades and so forth. Let's dive in a little bit on reliability because this is really sort of explains the core mechanisms underlying temporal. So suppose we have an agent. In this case, it's a customer service agent that's going to be helping with returns. And so this agent needs to connect to a whole bunch of different systems, right?

1:27 Knowledge bases, inventory, ERP. It needs to handle payments. It needs to probably have escalations. It needs to be able to send emails. And in order to make all of this work reliably, what do we need to do? We need to put retries in a whole bunch of different places. We probably need queuing to handle load. we need state management to keep track of the process all the way through. And these are all places where things can go sideways. And so whoever is building this needs to account for that one way or another. Now traditionally software interleaves the application concerns the business logic with the systems concerns things like reliability and the other systems concerns that we've talked about. Now fundamentally you know if there's one thing to take away from this talk about what temporal makes possible is it's this model of of durable execution. And what that does is it allows you to write code that puts the system logic over on one side and the application concerns on another and really isolate that out. Now the temporal open source project itself what it lets you do is it lets you build crash proof applications. It's open source license so MIT license and you can back it with a number of of different databases like Postgress or Cassandra. There's also a cloud product, but the key thing about it is that it lets you write regular code, regular programming languages, right? Python, TypeScript, you name it. and get crash proof execution. So, how do you recover after a crash? Well, there's basically two ways to get that state back. You can either save it or you can recomputee it. And with temporal, we give you, you can think of it, and we'll get more of this in the workshop, but simple annotations that you could put on the code, and you could say, "Here's what I want you to recomputee. here's where I want you to save state. Okay.

3:16 And then that's sort of if there's a trick, that's the trick. there's other cool things like distributed systems from ordinary language primitives. This is pretty cool. but I want to talk for a minute about how companies out there, our our users, our customers are using temporal. This is Venat VP of application infrastructure at OpenAI. And so one application that's notable is chatgpt images where there's a whole series of steps that need to happen. They need to all happen reliably in order to spit out that image. That was one of the the notable use cases that for temporal and open AI codecs on the web is another one. But really there's a whole slew of things ranging from the infrastructure control planes to the the data connectors that's effectively think about that as building and maintaining the the rag indexes and so forth. And then finally, traditional business processes. and really across the ecosystem, whether it's at Nvidia or at cursor now SpaceX or lovable repletoral is being used really across that stack to enable agentic AI. So now I want to dive in and just make a few observations about really sharing how we think about it and what we've learned from working with companies across the ecosystem. So we have this functionality that we're need to provide right it's an agent and that means different things to different people definitely in a business context it's basically some system that gets something done right and what we spend a lot of time thinking about particularly in the AI foundations team which is the team that I'm on is really what are the right abstractions what are the tradeoffs that come along with a sentic AI workload how do we slice them which ones do we pick a point in the space where do we maybe give you a knob for it. And so just a few observations about these workloads that I think are particularly interesting.

5:10 I'm not going to read through all of this, but you know just on the LLMs, right? So this is also I'll just say, you know, it's it's really it's a different different workload from other workloads we've needed to deal with before in systems. And so we really it's really fun to be able to sort of rethink these abstractions. So non-determinism, right? So that means that you call into an LM, you're going to need to validate and store those responses. but it's also interesting because it has this flip side where it's also sort of fuzzy and it's actually forgiving with respect to its inputs.

5:38 That's the sort of other side of the coin from its outputs not being necessarily 100% determined security. This is interesting from the threats side. So you often think about adversarial inputs, things like prompt injection and alignment, but also a great threat model when you're thinking about agents is is actually just bad judgment, right? because they like it happens that RM-RF and so that's driving a lot of our by accident right not not thinking that's driving a lot of our design considerations and then certainly from the execution profile the fact that you're interacting with a world that's a place where you want to capture your state you want to make sure that that you're there you know writing things down carefully and also bursty load which calls for serverless and so forth now another observation that we've made from working with companies throughout the ecosystem is that there's really this spectrum of agency and that the right solution for a particular application for a particular functionality could come from anywhere across the spectrum and actually the more agency you have it's not always a better thing. So when people think about agents oftent times they're thinking about this sort of level three which is an you know an agentic loop the LLM makes all of the decisions about what happens next. But if you go over to the left, you look at level one, there's actually a huge number of applications something like summarization or so forth where the job actually gets done really well by having regular code drive the control flow and having those LLMs inserted at very specific points, right?

7:13 Could be very reliable and if that gets the job done, then that's probably the right solution. On the other end of the spectrum, we go as far as self-evolving agents, which we now see in in some of the harnesses. And so there's there's a reason for that, but you can also think that's probably not also the the right solution for the job. And when we think about abstractions at Temporal Abstractions for Agentic AI, we're really thinking about being able to allow you to dial that in and to frankly go back and forth along this spectrum without rewriting your entire application. And so one thing that we are sharing today is the temporal agent harness. So this is not yet a supported product, but if you go and you start the the GitHub repo, then hopefully it will be soon. I'll put up a QR code in a second to that as well. but this is a set of abstractions built on top of temporal's core durable execution abstractions that make it easy to build practical agents for things that you know, whatever it is that you want to build. So we give you that ability to plug in the inner harness, which could be a coding agent like open code. could be something like open eye agents SDK which also has a lot of interesting sandboxing functionality pedantic any of these we've really worked hard to embrace this ecosystem you can plug your tools and again this is really designed to give you abstractions to mix and match whether that means you want to talk to it through Slack through teams you name it so this is just a a quick screen grab of what was an open code session I was in there just fixing a bug in a simple program and because I've run that with the agent harness. Not only am I getting the the durability, the reliability if I wanted to run this at scale, but I also get sort of for free, but frankly it's part of the package. I get the observability, the ability to dive in and see what was the agent reasoning about, you know, at at the various steps, what were the tool calls and human approvals and so forth that happened along the way. And that's the benefit that you get from really being thoughtful about the abstractions that you're bringing into to your agentic AI. So with that I'm finished here. I welcome you to stay in this room for the next session which is going to be a workshop where we're going to get hands-on with temporal. Thank you.

9:31 >>