Agentic UX: Three Ways to Let an Agent Build Its Own UI - Tyler Slaton, CopilotKit

0:00 I'm going to be talking about a genic UX today and if you're not familiar with what that is or why anyone would talk about it, you're in the right place. specifically we'll be talking about a protocol that we maintain called the AGUI protocol and how we see it redefining AI driven user experience. First though, I'm Tyler Slaten. I have been at Copilotkit for about 2 years. I lead open source there. I'm from the United States. I was born and raised in North Carolina.

0:28 So, at Copilotkit we have thesis. And the thesis is that all UI will be AI. What does that mean though? so if we look at this chart, we have our agent in the middle. If we think about the way that we're interacting with applications today, core applications like Figma, like Linear, like Stripe. They're becoming agentic, right? And that is our thesis is that this is going to continue to grow and the adoption of AI is going to be through UI. We're building all of these great agents, but how we actually ship them to users is through user interfaces, right?

1:08 And if we can kind of take a step back and look at where we're at right now, we send an agent a message and we get a bunch of text back. We're kind of in a similar situation that we were in the 1980s, you know. I wasn't alive in the 1980s, but you know, I can certainly read a command line and know that it's a lot different than going and clicking a button, right? >> really simple days, actually.

1:30 >> Those are the simple days, he says. >> >> Hey, maybe. Maybe. Maybe we're going into the complicated days. No, I'm joking. but I think that what we believe at Copilotkit is that we're entering the kind of Mac era, the user interface era for AI. And we're really trying to enable that as a company. But first, what is Copilotkit? Copilotkit is the front end stack for AI agents. We think of front end stack as anywhere that you're building a front end. This can be inside of Slack, this can be inside of React, Angular, mobile apps through Kotlin, Dart, you know, wherever you're building.

2:05 we build generative UI, which we'll get into. We allow you to provide human control for your agents, so collaborative interactions. And we do this at every surface. And a surface is where we do core work like Slack, but also in React if you want it but in your application. We created AGUI. AGUI, if you haven't heard of it, is the agent user interaction protocol. We'll be talking a lot about that as well. And everything that we build is open source.

2:27 so, we are currently at 35,000 GitHub stars, 14,000 in AGUI, and we actually have 3 million weekly installs across our packages. We also power 30 million weekly agent interactions. So, this is basically one turn of an agent. So, if you send a message, we facilitate 30 million of those every week. And we're used by for 50% of the Fortune 500s. and if we think about UI and why all of the UI that we look at today is text-based, it's because building agentic UX is really hard.

3:00 they break the traditional request and response paradigm that we've been operating on for the for the last 30 years of the internet, which is we send a request and we get a response in a timely manner. Agents kind of break that a little bit, right? They take a long time to do work, and so we have to re-architect our systems around this, right? Think about if you've ever built an agent that runs for more than 15 minutes and you're in a serverless environment if you have interacted with that before.

3:23 so, they're long-running. they also do structured and unstructured inputs and outputs. This is why they're flexible. This is why they're This is partly why they're useful, but it's also one of the things that makes it hard to maintain and interact with. We also now are bringing in this notion of composition, so we'll decompose a task and send that to different sub-agents to complete that task, and we need ways to show that UI efficiently. And they can also, as we'll see, build non-deterministic UI. And, you know, how do we control that? And how do you make that a nice user so that's why we built AGUI. it was actually to solve these problems.

3:57 And so AGUI is the agent user interaction protocol. It's for connecting agentic front ends to our agentic back ends. And we're going to be talking a lot about Mostra today, obviously. and lots of TypeScript. That's the beauty of TypeScript. Everything is going to be end-to-end TypeScript with React. But if we take a look at the protocol landscape, we just heard a great talk about MCP and where MCP is heading. MCP is for tools and context and with some recent additions, they also now can return iframes. So, you can embed whole app entire applications into the places that our agents are working.

4:31 A2A is basically two agents communicating to each other. So, if you have a you know, standard microservice or standard microservice architecture inside of a large enterprise and you have this kind of mesh of services, A2A is to kind of facilitate that at the agent level, right? So, the protocol is trying to enable that. What AGUI is trying to do is make it so that you can take that exact same agent or any agent in that mesh, insert it at any surface. So, Slack and React, WhatsApp, we have lots of different integrations.

5:00 and this is this is a pretty big chart, but I'm going to explain it very briefly. Basically, Copilot Kit is your front end. AGUI is what's connecting it to your back end. That's the TLDR of this entire diagram. And when we released AGUI, it really caused an ecosystem-wide shift. we've been working a lot with some of the hyperscaler enterprises like Google and Amazon and Microsoft and Oracle to ship AGUI at scale within their platforms. we are also adopted by many of the AI leaders in the start the AI space, Mostra included. And we work with a lot of consultancies to ship AGUI into production with a lot of companies through the consultancies.

5:41 So, how does AGUI work? Well, AGUI is a set of 16 standard events. We just added two recently for encrypted reasoning, so 18 standard events. and they're basically for an abstraction of agents. So, we created model routers like 2 years ago. That was on the rise about 2 years ago. This take took all of the different model APIs and condensed them into one unified stream so that we don't have to change our business logic to change out our models.

6:05 But, we never actually did that for agents. Agents actually have the exact same problem where when we're building different agents in different frameworks, now we have different streams, right? So, now we have the exact same problem that model router was solving. AGUI solves this. That's the entire purpose is to unify these streams into kind of one output. We can do lots of really cool things with this, but let's let's let's take a look at example run, right?

6:24 It basically boils down to just JSON data, right? that's the beauty of it is it's really simple and it's event-based. These events are just JSON objects. so, we can port it really easily to different frameworks. so, the adoption can be really easy. It's just send your events in this format and you have adopted AGUI. So, if you kind of go through these, it's the usual suspects. So, we have text message that starts it starts streaming content. We're delta-based protocol. So, that means that we'll only send the changes. so, you can create stre- efficient streaming.

6:56 Now, tool call will start. We also have this concept of state snapshots. So, I haven't actually been watching the talks, but if we've talked about memory, Monster has fantastic memory. It's one of my favorite parts of the framework. state is effectively for memory. Ways to surface the agent's memory to your users. It's not necessarily a part of the conversation, but it is a part of the conversation behind the scenes, right? So, it's a nice way to surface something to user.

7:18 and if we take a look at the diagram here, I really just want to showcase this is our agent backends. It talks AGUI out to our our clients. Our client exists across a bunch of different SDKs, and then that is how we actually ship across many surfaces. But, the really meat of what I wanted to get to is putting this all together live with you as though we are all working towards a solution to build some H and T Q X together.

7:44 And what we're going to be using throughout this talk is generative UI is going to be our design space. So, I want to introduce you to that design space. What is generative UI? Well, generative UI at Copilot Kit we kind of coined this idea that it's a spectrum. Generative UI is a way for agents to surface content to users that is not just text, right? Markdown has some like ways to do tables and kind of like small components, but this is like true reactive interactive UI. And to get this we have kind of three notches on a spectrum and a bunch of little tiny bits that kind of fit on that spectrum. So, we're going to be talking about all of these, but to introduce them, control generative UI is where we have a lot of control over the output of an agent and the agent just decides what it's calling. It's not actually generated on the fly. This is effectively calling a tool and mapping that tool to a component on the UI.

8:35 Declarative UI is where we're taking it one step further and we're composing a bunch of components together into one generation. And we'll look a little bit more into that and if you've heard of JSON render, JSON render is a great example. If you've heard of AUI, AUI is a great example of this. And then all the way at the end we have MCP ops, which is basically like a way to bring MCP ops into your or sorry, to bring MCP enabled applications into embedded chat experiences, which is pretty cool. And then all the way at the end we have open-ended generative UI, which is basically where we spin up some sandbox, we generate some UI and we show that UI to the user progressively, which we'll see.

9:13 So, I want to start with our agents. There's going to be a lot of code here, so I'm just going to zoom in a little bit, but I'm going to kind of walk you through it. So, I have a pretty simple agent here. I defined some incident tools. This is we're going to be building an incident triage bot. A lot of this is I'm just making a really simple one for the sake of demonstration. These tools effectively just return, "Hey, there's an incident and here's what it looks like." We're going to be making this pretty cool as we go along though. I use GPT-4-1 here just for speed and cost. Then I gave it some pretty simple instructions, nothing too crazy, but this is all the monster that you know and love.

9:44 This is all on the front end now. Then I provide a Copilot chat component. This is basically going to be a default way to render some interaction with the user. this is all in React, but it again scales to any surface. And if you're wondering what Slack looks like, don't worry, I'll show you. But you'll also see right there it says there's a runtime URL. This is basically a proxy to your agent that says, "Hey, I'm using AGI. Please use this URL for AGI."

10:07 Okay, so now we want to provide a component so that as the agent is triaging an incident, it can show us something about it, right? In this case I want to see the health. We have an incident where we're no longer able to check out code from our repository. So when we do get check out, that's no longer working. Why is what we're trying to figure out. So we're going to provide it a component where it shows us the health. Just a standard component. You can see here, we're going to parse the props that are coming out of it, and then we're going to pass that to the component that I wrote, which is just standard React if you've ever written React.

10:39 All right, so this is our demo. This is going to be what we're going back and forth from. It's embedded right into our presentation here. I'm going to ask it to now check out failures, are check out failures spiking, can you investigate it? And just like that, it returned that card that I just specified, right? It's populating all that data through the tool call, which you see right here. What's cool about this is the client is actually defining this tool, sending it through AGI to the monster agent, and the monster agent was never changed, right? So now the monster agent is able to know how to render UI inside of this surface.

11:10 Which is really important at scale because I can do this exact same behavior, for example, Block Kit. If you're familiar with Block Kit, I can now give my agent Block Kit from the Slack channel, which I'll talk about in a second, and it's able to render all of this UI. All right. Now I want to move on to declarative UI. So this is where we're going to basically generate a schema, let the agent generate a schema, and we're going to map that to some renders. ATUI is a protocol that we helped build with Google. We own the React portion of this. And we built a lot of these SDKs, or sorry, a lot of the React SDK that you see here.

11:48 So, first we're going to create a catalog. A catalog is a composition of two things. One is the actual JSON schema that I just mentioned, and two is actually going to be the renders, how we render that JSON schema. and then we just pass it into the Copilot Kit provider. If you're familiar with React, this is just a provider to provide context to an application. if we look at the catalog, who here is familiar with Zod? Any Any Zod lovers out there? Oh, nice. I love TypeScript AI.

12:15 so, I don't have to explain Zod, but we just define some Zod objects, which allows us to then pass this to the agents. again, the agent doesn't actually have to be aware of how ATUI works. ATUI is actually transporting the ability to generate ATUI to the agent. So, the client can define, "Hey, here's all of my ATUI." And then the agent can just say, "Great, I'm going to go use that." And you don't have to actually change any of your business logic.

12:38 On the other side of this, we just define a renderer, right? So, we define a severity banner and what it is going to generate here, and then we define what that actually looks like. So, we map the schema definition to the schema renderer. And if we put all of that together, I can actually ask it to build the incident workspace. So, build a workspace for this incident so that I can get more information about it. It'll take a little bit of time, which is one of the cons actually of declarative UI.

13:02 Not too long, though. And now I have this UI. What it did here is pretty interesting, though, if we really think about it, because I didn't actually specify how to put these pieces together. I just said that these pieces exist, please figure out how they go together. So, if I ask this to do this three times, I might get different layouts every time, or if I slightly change my prompt, it might change a little bit, but the components will always look the same. And that's the beauty of declarative UI is you can get really a set or you can compose subsets out of a superset of your components and compose those together. So, in this case it decided, "Hey, this is a severe incident.

13:37 Here's why. Here's my evidence. Here's my hypotheses of why I think this is happening." Which is pretty cool. I mean, it didn't really do a whole lot here. Really, we just specified, "Hey, here's the data that you can generate. Here are the components that you can build. Go and build it." And I gave it a pretty fuzzy prompt here. If we saw the agent from the beginning, you can see here on the left there's a whole lot going on here.

13:56 All these tools do has basically tool for getting data from API. So, it gets the data from the API, notices that an incident is occurring, and then it's calling these components back to us. So, now I want to talk about open generate UI. All I do is set open generate UI to true. And again, through the AGI abstraction, it's going to transport tools from the front end to the agents. And the agent's going to know how to build sandbox UI on the other side.

14:24 So, in this part of the demo I want to make it super clear. I did not ask it or I didn't specify what this component looks like. It's just going to go generate it kind of on the fly. So, hopefully this looks cool. It is totally up to the agents and the intelligent model behind it to come up with something that is relevant to the user. You can see it generate live here. This is what it came up with.

14:43 What's cool here is I did ask it kind of in some of my prompting behind the scenes to give me a little button called review rollback. And so, what that's actually going to do is render a new card where it says, "Hey, I want to roll this back. Are you going to let me?" How do we do that though? So, if you're familiar with Mostra, Mostra has a concept of suspend and resume. The AGI protocol has the ability to interact with these suspend and resume type of workflows. So, I'm just going to show you what that looks like. First, I create a tool. I give it a suspend and resume schema. And then inside of the execution of that tool, I just grab the the resume data and I call suspend.

15:23 so at this point when suspend happens the agent is no longer executing and then we will call it back with the A GUI protocol to reactivate it basically. On the UI side we have a hook called use interrupt which lets you just define, "Hey, when an interrupted happens, please render this UI and here's how you resolve it." So we have this little resolution handler here to resolve that. Then I can go come in here and I can improve that rollback and it is going to resume the agent and the agent is going to say, "Great, the incident's resolved.

15:51 Yay, awesome." But I have been talking a lot about Slack. So how does this all look inside of Slack? Well, we have what we call our channels SDK. So the channels SDK is a super set of all the different messaging platforms that we know and love like Slack and Teams and WhatsApp. I'm going to be focused on Slack today. One of the things I didn't show you is actually how we connect through A GUI to our surfaces. It's actually as simple as just providing the URL to the master agent. So I define an HTTP agent. This is coming from the A GUI client.

16:24 I provide the URL and voila, I have A GUI, right? Then I can create a channel through the channels SDK. in this case I'm going to name it Kite. Copilot Kit's logo is a kite, that's why it's called that. And so when I go into Slack and I type at Kite, it's going to trigger my bot. I provide the Slack adapter with a token and app token. This is just some trios that you need to do behind the scenes. And I also provide that agent that we just defined.

16:48 So the same agent that we've been using this whole time inside of Slack. I also provide it with some tools. So earlier I kind of promised, "Hey, I can also define the ability to render block kit." That's actually what the default Slack tools are. So we use A GUI to generate the block kit schema and then pass that back to Slack and voila, we get we get block kit. If you don't know what block kit is, it's kind of like a declarative UI language for Slack and so it looks a little bit like this. so I took a previous execution of this and I basically said, "Can you explain what's happening here?" just took a screenshot of that generation that we got before with ATUI and I said, "Can you explain it?" And Kite responded with, "Hey, here's what's happening, right? this is a block kit component it just decided to render, but I can actually also say at Kite's, "Hey, I don't actually understand this.

17:37 Can you show me some UI maybe a flowchart of what happened?" And that's going to instantly trigger it through that code that I showed you. So, the agent is going to start thinking. You can see that happen right there in inside of the thread. If we open it up, you'll see Slack recently shipped this really nice like agent-first experience where you'll see this nice thinking UI as it's executing. It also shipped some really cool like streaming printouts. So, you saw like kind of like how our chat looked like a second ago is in Slack as well. And you actually have the ability to open up full chats. You can like DM this and interact with different past conversations. You can bring in to your threads wherever you're doing your core work. And all of this also works for teams, which is pretty cool.

18:20 But what's happening here is you can see it generated this like image, right? What's actually happening here? it created this big complex flowchart which is pretty fun, but famously a block kit is only a subset of all UI that we can build in the world. It has like, you know, the ability to render cards. It has some buttons that we can render. We don't have too much control over it. You know, we can just the color and stuff like that. What we're actually doing here is we're using something we call channels UI to spin up a headless browser with a reactive components and take a screenshot of it and then send that back to the client. So, you can actually use any component that you've already written inside of React or Angular. You can spin it up, take a screenshot of it and send it back without ever editing your actual any of the UI that we just built a second ago.

19:06 Which is pretty cool. And then also we have the ability to like specify behaviors. So, in this case this only triggers when I mention it. So, if I say @Kite, do this. and then I can call thread.run_agent, and that's going to take care of all the logic of actually calling my agent behind the scenes. So, if we recap what we just did, we just took a monster agent, we gave it a React UI, built a pretty simple triage agent where we were able to visualize our incidents, get a dashboard for it. it acts asked us if we want to roll back, and we said yes, and then it was able to resolve the incident kind of in the flow with interactive components as opposed to us just typing.

19:44 our next thing that we're building and one of the cooler things that we find from all of this is all of this these threads, these interactions with a user are really great signals for learning. So, if you've used something who actually I like to do this. Who here uses Cursor still? Okay. Who here uses Claude? Okay, a lot more Claude. Anyone a Codex fan? I'm a big Codex guy. I actually use something called Composer or Conductor, which lets me use all three, but all of these harnesses are actually using your interactions with the harnesses themselves to improve the harnesses and also the models.

20:22 So, this is what's happening with like GPT-5 3 Codex when that came out, Cursor took all of your interactions that we got from the last couple years and built the I think they call it the Composer model, if I Yeah, is that right? Thanks, everyone. Nice community. and same thing with Claude. They're all kind of like self-improving from the interactions that you were giving them. which is really cool. You can basically get this like recursive loop of improvement on your models because us interacting with them are really great signals for how to improve them.

20:57 So, what's also nice about agent UI is this is the same concept that they're all doing, but abstracted into a single set of events. And we also have the ability to get really well annotated human feedback from these interactions, right? I clicked approve. I could could have also clicked deny. and you know, I won't get into the specifics, but you can get like kind of everywhere in between. You can have muddy and complex interactions and you can annotate those really well because you're interacting with UI as opposed to just text, right? You don't have to do any like sentiment analysis on that.

21:29 So, what we're building right now is what we call the self-learning SDK, which is basically where we're going to take all of these interactions, we're going to trace them, create evals against them, generate skills for your agents, and then insert those skills into the agents. That's something that we're currently building. So, if you're interested in that, come talk with me after. That's been my talk. Our whole thesis is to let users use your use your your agent anywhere. if you want to talk with us, you can scan this. that'll get a link for you to book some time with our engineers. If you want to get started with any of this, you can run MBX CopilotKit create, and that'll actually let you get started with a mastery app right now. So, thank you for your time.

22:09 >> >> You mentioned that people could find you after, but how about we take some questions now? >> Yeah, absolutely. Let's do it. >> Thank you very much. Thanks for the for the talk. It was really cool. I think that generative AI is is the future. do you have examples or experience with generative AI outside the chat? You know, like you basically the AI provides the user interface, but you never It's more like a wizard that dynamically changes. What's your experience there?

22:41 >> Yeah, so one of the great parts about working at CopilotKit, one of the things I love working here is that we get to see all of the different types of interfaces that people are trying to come up with. And I think that if I had to put it into kind of like two buckets, it's what we were calling workflows a year ago. I think it changes every day. and chat, right? And workflows are really like when I click a button and I trigger some AI, what does that look like inside of the UI? And that has a lot of really interesting implications, right? I think that one of the biggest ones that I've seen that I really like is search. Search fun can be agentic, right? And we can have really great agentic UX around that. So like, if I go into a website and I type Actually, if I go Google is a great example of this. If I go to Google and I type like, "How do I catch a salmon in Stardew Valley?" And it comes back to me with some results. The first thing you see is actually an agent result, and I can start to chat with that at the end of it, but it doesn't necessarily mean that it is a chat-driven experience. I was just asking a question, right? That could be chat, or that could just be, "Hey, here's a response and you never think about it again."

23:46 That's kind of like the table stakes of non-chat. It still kind of is like chat, right? Like I'm still typing the message and I'm getting a response. But, some other stuff I've seen is like, you saw I was saying like, "Hey, there's an incident, right? Go and involve it." you can actually click a button and do that instead, and that goes and triggers a workflow. And I think that Monster has some really awesome concepts of workflow where you can build some really nice directed graphs to get to some end state.

24:12 And what the great the great news is is that all of these events are generic. They're not necessarily result specific to chat. You know, models take input and send structured or unstructured outputs out, so text or JSON. we can map that to any UI that we want. So, one of the demos I didn't show here is something that we usually refer to as a shared state. I kind of glossed over it a little bit. so Monster has some really awesome concepts of working memory, which lets you your agent basically update its memory on the fly and do that in a structured way. AGI actually lets you see that on the client and actually also update it bi-directionally, which is pretty cool.

24:47 So, your agent can see the memory and update it. So, if we put all these together, the common use case that we usually showcase is like a to-do app, right? Where I go and mark some to-dos, and maybe I open it chat or maybe ask the chat, "Hey, make a bunch of to do's." and it can literally update the entire UI as opposed to just living inside of the chat. I have a lot of cool stories about that if you want to hear more. I'd love to get some more questions in.

25:09 >> you mentioned tool call approvals as one form of feedback. Is it possible to define my own type of way to capture feedback like a thumbs down on every message and that feels Is that possible? >> It is absolutely possible. You're asking in the self-learning context? >> Yeah, yeah, exactly. >> Absolutely, yeah, it is. I think that thumbs up, thumbs down is kind of table stakes when it comes to self-learning. I think one of the things that ourself our our layer of the stack is uniquely enabled to do is provide you with even better like non-opted in cuz you have to think about that that's that's elective that somebody gave you a thumbs up or thumbs down. I never thumbed up or thumbed down my chat responses. I don't know if you do.

25:47 and so that's elective, right? So you're only getting like a biased set of information. What's nice about this is the default flow of how to interact with your agent is what is being trained on, right? That's one of the nice benefits of it. But the answer is yes, you can absolutely do that. >> So, thank you. >> there's a question here. Thank you. I have to say it's very cool being on this stage with you after we did this in That's That's a few months ago and now doing it in London. Feels oddly familiar.

26:19 >> >> Yeah. >> And yeah, as usual some great questions. >> Yeah, thank you very much. question about mobile apps. Do you have any thinking about how you could do something similar for the mobile native applications? >> for mobile notifications, >> No, mobile native applications. >> Mobile native applications. well, first of all, we have you're already able to do this today in a couple of different ways. we maintain the React Native implementation of all this, so you can go and use React Native to build your cross-platform applications there.

26:49 but we also have a bunch of community members that came, you know, all of this is open source. Agy is an open source and open governed protocol and so we receive a lot of integrations and requests for different SDKs all the time. So a couple of the cool ones I've seen is Kotlin was recently contributed. We had a dot net SDK so you can go and build native apps on Windows. That was actually contributed by Microsoft which is really awesome to see.

27:13 We recently also received Dart if you use Dart. The one thing that we're still trying to get is Swift. So if you are a Swift developer I would love to see a contribution from you. Or if anyone is. I'm certainly not though. But hopefully that answers your question. >> Amazing. Thank you so much Tyler. >>