Day 6: Harness Engineering - The Agent Environment

0:00 So, in this video, we're going to be looking at the fault harness component. The environment. Yeah, the environment. So, you have a couple of tools, but where will these tools operate? Where would they operate? Which runtime would they operate in? Which file system would this read file and write file tool read and write to? Which shell would this bash tool execute? Which network would this fetch page reach, which what network will it have access to? Which database resources is this query DB allowed to query? This is what the environment answers. Because all these tools are useless without an environment to operate in. Yeah, they're all useless without an environment to operate in.

0:44 Every tool with side effects needs a target for these side effects. That target is the environment. Every tool has to point somewhere, has to point to something, has to create what it needs to create inside something. And that's basically what the environment is. It's not a big lesson, it's just that simple. The read file and write file tools need a file system. mkdir also needs a file system. Where fetch web page needs a network and even where to dump its results. query DB needs a database to read in and bash needs a shell to run in. And the harness is responsible for providing that environment. So, what does a good environment look like? Let's look at what a good design for an environment in the harness looks like.

1:26 The first design requirement is that it has to be a bounded environment. Yeah, the environment has to be bounded. You can't just allow your tools to run wild in any type of environment they want or on your entire system. What if they want to rmrf some very important files from your drive or even your entire drive? What if a network tool starts to exfiltrate data from your system? It just sends it to some sort of command what do they call that thing? Control and command server.

1:56 And that just consumes all your sensitive information. Or what if you just have this tool that is calling a service and just burning all your cloud credits? It's burning all your cloud credits without any form of threshold, without any form of limitations that have been set that have been configured. Yeah, you need to define a scope. You need to define a scope for your environment. Your file system needs to have a scope, meaning that you have to restrict it to a particular workspace.

2:20 And that workspace definitely should not be all of my machine. your search page should actually be restricted to certain websites, certain networks. It shouldn't just be able to go anywhere. You kind of have a white list or a black list at least. So that it doesn't just go to malicious sites and just download malware. Also, you have to point your database to a specific database. Everything has to be bounded. Everything has to be bounded. Even your database tool probably should have the kind of operations it can perform. Probably shouldn't be able to delete data. That should be something you probably should do because we keep seeing agents deleting data these days. And yeah, if you control that from the prompt angle and you don't have the harness implementing it, then you're going to face some serious problems. Later on we're going to be seeing something called soft constraints and hard constraints. And soft constraints are basically instructions you gave to your model. Don't do this. Don't do that. And all those type of things. Like, oh, don't do this. Don't do that. Those are soft constraints. And why they're called soft is that your model can ignore them.

3:26 Your model can ignore them. Tell your model a thousand times don't delete my passwords file in my Linux environment, but your harness is giving it a delete tool that can do anything that doesn't have a bounded environment. It can ignore that instruction. If it feels if it deems it okay, it can just ignore that instruction. So your harness has to be the one enforcing that constraint. And that's where hard constraints come from. Hard constraints are constraints that are enforced by the harness. Soft constraints are basically instructions you're giving to your model and they can simply ignore it. So, that's one thing you first have to realize. By the way, I realized that this is environment environment sorry environment. Good. So, have a bounded environment. Very very important. Restrict the scope of where your tools can operate. That target restrict the scope of the blast radius of your tools. Number two, your environment should be reproducible.

4:19 Yeah. You should be able to reproduce your environment. So, if you start a session and you want to build, let's say, an hotel management application, we should have a clean new environment. A clean new environment should be created for that. Every run or rather every task should start from a clean environment. Every task should have its own clean environment created for it. You should not reuse environment across different tasks. You should not use reuse it across different tasks. I'm not talking about one task where you're going multiple times or multiple times.

4:52 I'm not talking of one particular session. No, a particular task like clicking new chat on your chatbot. We all use chatbot so that's something that you can you're familiar with. If you click new chat, it should not bring you cannot just dump stuff from the previous chat. No, it should not do that. Later in memory when we discuss memory, we can see how to pass information between task. But, when it comes to environment, you create a clean new environment.

5:17 That's why when you create click new task, you'll see a blank chat. So, that's that's the mental model you should have for your environment. Whenever you want to start a new task, you should create a new environment. You should not inherit leftover files, mutated databases, or dirty processes from previous task. No. Every task should have its own dedicated environment. And this way, when something goes wrong, you can rerun it and see the same failure. And when you see that failure, you know that this failure is not from the environment.

5:45 It's not because I had leftover files in my environment from previous tasks or something bled into my context. No, this is basically coming from the model and the agent itself. Also, your environment should be inspectable. Yeah, we should be able to inspect it. As an engineer, you should be able to see what the agent did to your environment after a run. That is what files it created, what command it ran, the processes that were spawned, and stuff like that. Those things are very, very important, especially when we get to creating our observability framework inside our harness. So, your environment has to be inspectable. We should be able to see stuff, see the processes, see the files created, see the folders created. Every side effect to your environment should be visible. Now that we have a good understanding of our environment, let us do our case study. I'm sure you'll love those.

6:33 And this time we're going to be looking at Codex. Yeah, we're going to be looking at how Codex manages its environment. Now, one brilliant thing that Codex does is that it runs every task in a sandbox container. It runs every task in a sandbox container. So, what it basically does is that it clones your repo and puts it in a sandbox container. Now, inside this container, the agent can read, it can edit, it can run tests, and even execute any arbitrary code without touching your real machine. Everything is done safely inside the sandbox container. And when the task is done, the container will be destroyed. This container will be destroyed, and changes will be sent to your environment as a pull request.

7:19 Changes will be sent back to your environment as a pull request. So, you're only getting what worked. You're not getting all the chaos that happened inside the sandbox container. This is why engineers trust Codex to run any code they didn't write. You can just pull anything, dump it inside Codex, run it. Codex is not going to run it on your actual machine. It's going to create that sandbox, run it inside. You will know whatever is happening with that code inside that sandbox. The sandbox will deal with any type of chaos or any type of malicious intent that might be coming from that type of code. And you don't have to deal with all that. And you can choose to accept it or reject it, keeping your own environment safe.

7:56 You can even perform experiments inside that sandbox. You can decide to delete some files and see their effect. You can run wild bash commands that you probably didn't write, something you copied from Stack Overflow or even copied from something like ChatGPT. You can also hit the network. You can just browse some very very strange sites. Everything can be done safely inside that sandbox because the blast radius of any type of threat has been bounded to a disposable container. Yeah, it has been bounded to this disposable container. Without the sandbox, the same agent doing the same work will be touching your real file system. All your credentials, all your repos, everything. It will be touching them. And I don't need to tell you how dangerous that can be. But with Codex, something that we all fear before and avoid, that's running arbitrary code, it's just takes it from a reckless idea and it becomes something that is an engineering practice. Like, "Oh, yeah, let's see how this blows up." And if we see that it blows up, we'll just destroy this and come back to sanity.

9:00 So, yeah. That's the beauty of Codex. That's the beauty of how Codex handles its environment. In the chapter dedicated to the environment, we're going to also see how we can use Docker as our box and we can just run stuff inside the Docker container instead of running it on our actual system. How we can use Docker as some sort of sandbox environment.