How to use Karpathy's Autoresearch to 10x Claude

0:00 Andre Karpathy, one of the leading AI researchers, recently launched Auto Research, a framework that literally makes AI become self-improving. And when applied well, it lets your AI agents and your skills autonomously optimize itself against real-world data to optimize anything from cold email campaigns to landing page conversions to content engagement or anything else that can be measured. So, in this video, I'll show you how to actually use it with best practices, show you how it works with plenty of examples, and a simple way you can get started with this in Cloud Co-work or Cloud Code today. So, Karpathy introduced this Auto Research concept or framework a few weeks ago, and he built it specifically for machine learning, but this can be applied to many other concepts in AI, too. We, for example, built our own version of this framework with a plugin for Cloud Co-work and Cloud Code, and we've been using it to auto-improve our skills, our Cloud MD to make our second brain more efficient, and we're now testing it on our newsletter subject lines to improve open rates, LinkedIn posts to increase engagement, and landing page copy to improve the CTR. Of course, this framework could be applied to many other things. Now, before showing you exactly how it works, let me show you some examples. So, I'll start with a simple one so you understand the concept. So, here I ran the Auto Research skill that we created to optimize my LinkedIn writer skill for the following criteria: optimize the LinkedIn skill to make sure hooks are always under 136 characters, and make sure bullet points are always done with hyphens. It then ran one test to identify the baseline score, which in this case was 80% or four out of five that passed the two criteria, and it then started the optimization loop with a hypothesis on how it could improve the baseline. So, in this case, it started with a hypothesis on how to improve the hook criteria, in this case, add explicit 136 character limit for hooks in step four in the skill. It then ran the first iteration, and the pass rate went up to 100%, so it decided to keep the hypothesis because it was a significant improvement. It then ran iteration two to improve the second criteria with a new hypothesis.

1:59 So, in the end, it ran five iterations completely autonomously, and the final output was a 100% score with a 25% improvement for both the criteria. And of course, the new optimized LinkedIn writer skill and a dashboard where we can see all the iterations that it ran, which ones it kept, which ones it discarded, and also the overall improvement from where we started. Now, this was a simple optimization, of course, but we can make this much more extensive and in-depth. For example, here I ran it again on making sure it uses the correct hook templates according to my hook template reference file, and another criteria to make sure it adds nuance in my sentences when I make bold predictions or claims, another one to optimize the skill to always follow a proven writing framework like PAS or AIDA, and making sure that my post contains a personal story from my personal background reference file. It then again first established the baseline, which is 68%.

2:51 It started running iterations with hypothesis. It then ran 10 iteration loops completely autonomously to get to a 27% improvement from the baseline. Again, it gave me the new updated skill and a dashboard with the overview. Now, this can not only be used for skills. I've also used it to optimize my second brain performance. If you watched my last video, you'd understand this. For example, here I ran the loop on optimizing the Claude MD for knowledge routing to make sure it saves the data in the right folders, to optimize it for wiki link creation, and also for knowledge retrieval. Again, it ran five iterations and got to a 9.9% improvement and the updated Claude MD file. Now, coming up with a few of these criteria yourself and just listing them out is a great way to improve your skills or your Claude MD fast, but this becomes especially powerful when we start optimizing and running the loop against real-world data. Firstly, of course, we can do this based on past data. For example, in this chat, I pulled my top performing LinkedIn post and my worst performing LinkedIn post together with engagement data, and then I let Claude identify patterns to come up with a hypothesis or criteria on what to optimize the skill for based on this real-world data. So, did a LinkedIn pattern analysis comparing my top performing versus my worst performing ones and then compared a few things like hooks and the hook type versus metrics, the CTA analysis versus metrics, and what kind of writing framework was used versus the metrics.

4:13 Based on patterns it found in these three things, hooks, CTAs, and writing frameworks, it then created a new hypothesis on what kind of posts would work best. But, the next step is of course that we can start using a scheduled task together with auto research to potentially make it autonomously optimize it on a weekly basis. So, I've set up three of these tasks like the LinkedIn auto optimizer, the email sequence open rates optimizer, and auto landing page optimizer. Now, they haven't actually run because I started this this week, but the idea is of course that every week it can set an up hypothesis. For example, this combination of hook, writing framework, and call to action gets the best results. So, we'll test those types of posts. Then the week after it scrapes the data from the performance, keep the change if it led to better results, and keep iterating until it hits the goal.

5:00 For example, until it reaches an average of 250 average likes. Now, of course you can also build some human in the loop in here if you're not comfortable with an AI actually running this autonomously. But, for example, in the email sequence open rates and auto landing page optimizer where you can test for example H1 copy until it gets to a 3% CTR. And you can see in the instructions it's a set of steps. So, it first has to pull last week's landing page metrics, right?

5:23 Then define a new hypothesis for how we can improve the landing page or H1 copy, then make the change on my website, and do the same next week. Now, of course there will be other variables when you're testing this, so it's not going to be perfect, but you can see the potential for a framework or set up like this, especially with agents becoming more powerful and getting more context around our business. So, even if you're not ready to use it like this yet, it's still great to optimize your skills or Claude MD really fast. Now, before showing you how it works and how to use this efficiently, how do you actually set it up? Now, the way you can use Karpathy's framework is by going to his GitHub repository, which I'll make sure to link in the description below, too.

5:58 And then you can basically download the zip file. And once you've downloaded it, you can just unzip it. Then go go to Claude Co-worker Claude Code and just give it access to that file. And then once you have the folder selected, of course, Karpathy's framework is built for machine learning optimization, so you would have to go through it with Claude and adapt this to make it applicable and usable for skills, for example, or for the Claude MD or any task or process you want to optimize it for. So, you would basically just ask something to Claude like, "Can you apply this framework or loop to optimizing skills, etc." And then you can start building your own version of it. You would need to go back and forth a little bit to actually make this work efficiently.

6:39 This is how we built a plugin out of this that helps it optimize for the most common use cases. You can see we have a skill here with auto research and also the agents that do the evals, the judge, and our test runners. So, you can definitely create something like this for yourself if you go back and forth with Claude a little bit, but if you're interested in our plugin, you can also check out my AI accelerator, where we list all the plugins and skills we're building out for ourselves. We also have multiple weekly Q&As and AI workshops where we dive a lot deeper into these tools and unlimited one-on-one live tech help to help you set up anything you need in Claude Co-worker, Claude Code, or anywhere else. So, if that's interesting to you, definitely check it out in the first link in the description below. Now, whether you want to build this out for yourself or just use it, it's still important to understand how this actually works.

7:21 Now, it's actually pretty straightforward. All it is is just a loop. First, you define the criteria of what you want to optimize, for example, optimize my LinkedIn skills so that hooks must always be under 130 characters. Then auto research will first establish the baseline criteria by running a sub agent who tests the current version and gives a score. Usually, it runs a few tests, so for example five. You can see here that in Cohere work it used a sub agent here, the auto research test runner, to establish the baseline and give a score.

7:51 In this case, 80%. And then the main agent, the one you interact with on Cohere work, creates a hypothesis on how it can improve that score. For example, by adding a row in step four in the skill. Then another sub agent runs a test with the updated skill. Basically, in order to make sure it's unbiased. For example here, it started testing this hypothesis with another sub agent here. Then there's another sub agent that again is unbiased that evaluates the outcomes of each of the test runs. Now there are two ways you can evaluate the outcome. It can be through a script which just defines if something happened or not through deterministic logic or a Python script. Or it can judge the outcome through another sub agent if it's not possible to do it through a deterministic code script. For example, in this case it's a simple criteria, so the eval runs through a script. But if we look at another example where I tried to optimize the skill to match the hook with one of the hook templates that I have in my hook reference file, it actually is harder to do it through a script. So you can see here it first ran a sub agent to run the test, and then the outcome is not judged through a script but through another sub agent, which is the L&M judge, right, auto research judge. He judges the outcome if it passes or not. And then the main agent gets the results back and decides to keep the change or discard the change depending on if it improved upon the baseline score. It then keeps setting new hypotheses until it reaches the goal or it reaches the amount of iterations you've defined. Have you seen my previous video on skills 2.0 or evals, which is a built-in feature of the skill creator skill in Claude, the difference is that this basically automates the entire optimization process autonomously, while with evals we have to constantly judge the outcome of an eval and give it feedback for the next improvement. So instead of that, we'll just keep running in a loop until it actually meets your criteria. Now, of course, this comes with a cost, too, because if you let it run for hours in terms of optimization, you can imagine that the token cost will go up quite a lot. So, it's important to actually optimize things and skills that you use frequently and make sure you optimize it for things that make a real difference, which brings me to the next point, which is the importance of defining good criteria. Because the skill or the process you're optimizing with auto research will only become as good as the criteria you define.

10:06 So, what's an easy way you can think about formulating good criteria? First of all, whatever you're testing against, keep in mind that the outcome should always be true or false. That's how the system works. So, first, then you have to state the exact condition, not just the goal. So, don't make don't say something like, "Make sure the hook is short enough." Say, "The first line must be under 136 char- characters." So, generally, you need a specific number, a specific format, or a specific pattern you can actually test against. And then, second, of course, you need one variable per criteria. If you need the word and to describe what you're testing, then you have to split it into two criteria.

10:42 You can still give it multiple of these criteria at the same time because it will go through all of them, but each one should be well testable, so separate them into two. Now, how do you come up with these criteria? This, of course, becomes a lot easier if you actually have real-world data, experience, or expertise around the process, task, or the skill that you want to optimize. And an easy way to come up with some of these criteria is by either feeding it some of the real that real-world data, like I did with the LinkedIn post, and asking Claude to spot the patterns and come up with criteria to test against based on good and bad-performing posts.

11:14 But, even if you don't have that real-world data, just try to feed it good examples or good outputs and tell Claude to identify patterns in them and potentially criteria to improve the skill upon. Now, if you don't have that data, of course, you can also look at industry best practices or other people's posts or data. Now, there's also a big difference, of course, between optimizing for clear-cut deterministic skills or processes and subjective or creative skills or processes. For example, for an accounting skill that goes through my Stripe, my invoices, and logs it into my Excel sheet at the end, it's a pretty clear-cut deterministic process that just needs to work well. And there's not really any external data we can test it against. So, in those types of skills or processes, we can often just run an eval from the skill creator skill. If you haven't seen that video, I'll make sure to link in the description below, too.

12:02 To basically make sure that the skill actually works the way we intended the skill to work. But I think the real unlock is using auto research specifically to optimize processes and skills that have some subjectivity or creativity embedded. And if you think about it, most processes and skills across marketing, sales, customer support, or other business departments have some level of subject subjectivity embedded. But that, of course, brings up another question. If we have to give it these very clear true or false criteria for auto research search, how do we actually use it to optimize it for these more subjective or creative processes like a LinkedIn skill? For example, how do we optimize it to match our tone of voice? Now, if you think about it, even subjective or creative skills like copywriting can actually, for a large part, be defined by true or false criteria. And the way you can think about it is through this three-level framework. And the first level is what are the clear-cut rules it always has to follow according to your or general best practices. For example, on LinkedIn copywriting, that could be the hook must always be under 136 characters.

13:03 Second, the sentence sentence length should always be between 5 and 12 words. Or paragraph shouldn't consist of more than three sentences. These are kind of like hard rules and well-definable and general best practices for LinkedIn copywriting. But then the next layer becomes a bit more subjective, right? And this is where we start looking more for patterns of copy or your unique way of writing. For example, I tried to optimize my skill to make sure that bold predictions and claims are given with nuanced words like I think, I believe, or I predict because I noticed that AI makes very bold claims in a in a too convincing way. And for these types of tasks and optimization, we generally need that AI judge, right? Not just a script. Another example for that could be, for example, to make sure that your copywriting follows one of the major writing frameworks like PA S, AIDA, CPF, etc. And then for this layer, which is a bit more subjective, I think you AI can really help you spot those patterns or criteria I could optimize it for. Now, of course, there's always going to be some level of creativity or subjectivity in these types of skills that you cannot optimize it for together with Auto Research. But the point I'm making is that if you can articulate these rules or let AI analyze your post and identify these patterns or criteria, you can get it to match your tone of voice to a very high degree by just defining these true or false criteria. So, that's the way I generally recommend you optimize these skills or processes with together with Auto Research is by first thinking of those hard rules or best practices, run it through an EVA or an Auto Research, and make sure that it consistently nails those. And for some more deterministic skills like my accounting skill, for example, this would be all you need. And then for more creative skills, you can go to the more subjective layer while still trying to get those true or false criteria. And once you've done this, of course, the real meta is to start optimizing against real-world data. Now, of course, this becomes a bit trickier because there's a delay in the feedback loop, which you can solve with the scheduled task. But you have to be comfortable with AI autonomously doing this for you. And that's again why you need to make sure that the skill's already really well optimized before you go into this stage.

15:08 So, how do I actually start using it? Now, if you're going to use the plugin that we've built, you can just go into my AI Accelerator and download the zip file of the plugin. Once you have the zip file, you go to customize, you go here to plugins and click plus. You can click click create plugin here and upload the plugin where you upload the zip. All right, you just upload the zip and then it will appear here. As you can see, we have auto research. Now, mine was installed, of course, already. Then we have the skills. Here we have the auto research skill and three sub agents. One for the e-vows, one as the LLM judge, and one is the test runner.

15:40 Then you just go to a new task and you run the skill auto research. It will then ask you some questions if you haven't already specified what you want to use it for. So, in this case, I'd say optimize my LinkedIn writer skill. We've embedded in the plugin there also the roles for the criteria, all right, state the exact condition, one variable, etc. And then it actually proposed me some things to optimize for, but of course, here's where you want to start thinking, okay, what you want to optimize this for. Again, here's also the point where you can start feeding it real-world data or for example, let it analyze your top-performing LinkedIn posts. but let's say I just define four criteria here. I want to optimize the LinkedIn skill for these criteria.

16:22 So, I just added in four criteria. I let the skill do its work and it will already know what to do. Then it defines your criteria a little bit more in-depth. It's good to read them to actually make sure that they make sense. In my case, they do, so I just tell them they look good. And then it asked me which evaluation mode should we use? Now, this depends on if this is a really clear-cut rule or if it needs an AI or an LLM as a judge.

16:43 Now, you don't really have to make this decision. You can make the decision, but the system itself will give you a recommendation. In this case, for many of these would need AI as a judge. So, that's why it's already recommended that. So, we just click that and we can decide how many iterations we want to run this on. Now, generally, there's two ways you can do this is you can either define how many iteration loops it should run or you can tell it to keep running until it hits a certain goal or number. Now, generally, what I recommend, also to in terms of saving token usage, is to start with five iterations, maybe 10 if you have many, instead of letting it infinitely do a an iteration loop because in my experience if you let it run for too many iterations it actually gets worse after 10 or 15 iterations.

17:25 So, generally that's sort of the cut off that I recommend. So, in this case I'd I'd say five. It then scored the baseline results and prioritized the weakest areas, personal story specificity and nuance on both claims. So, that's the first iteration so we'll try to optimize. You can see that the first hypothesis it's testing is add explicit instruction in step five to always include personal anecdotes from Ben's background. And as you can see in this first iteration result it got an improvement on the personal story specificity. So, it decided to keep the change.

17:56 In the second iteration it then tried to optimize for nuance on both claims. It got a 0.6 improvement and again the result was keep. And it keeps doing this until it hits the five iterations. Now, this is going to keep looping until it does all five iterations. And then if you want to take this up a notch which I can't show you because it hasn't run yet is to use the schedule task. But in this case it would work something like this. You're running the weekly LinkedIn optimization loop for Ben once per now this is an automated AB testing system that changes one variable per week hook type writing framework or CTA CTA type to maximize average like post target 250 average likes. So, we just give it some context on the folder structure because we have to save some data. so, it actually understands what tests we have run in the past. So, first it's reading that context to get the current active experiment details to read the optimization rules to understand allowed hooks, frameworks and CTAs. Then of course scrape last weeks posts, extract the engagement metrics, calculate the metrics, update the context so we save that in the folder.

18:57 And then of course make the decision based on the results. If the average likes went up versus baseline keep. Average likes are down revert. And then generate the next iteration or hypothesis which will then be of course scheduled for next week. Now, I'll do another video on this when I've experimented a bit more with these scheduled or autonomous optimization loops, but I think it's interesting to experiment with. Now, that's it for this video. Again, if you want access to this plugin on all the other skills and plugins that we're building out, you can check out my AI accelerator in the link in the description below. we also do these weekly Q&As, multiple weekly Q&As, workshops, and unlimited one-on-one live tech help, was have a full guide on setting up your AI operating system or your second brain. so, if that's interesting to you, you can check it out in the link in the description below. Also, if you want to learn more about Claude and skills, etc., you can check out the video here above.