From Vibes to Evals: Building an AI Writing Tool That Works

0:00 Most people build AI products on by, tweak a prompt, eyeball the output, and just ship it. This works until your system gets more complicated, more features and edge cases, and then you can't tell if the changes help or hurt. In this session, Isaac shows how he hit a wall building an AI writing tool, and how he moved from vibes to structured eval. And that's why we're having Isaac present his work. It started with kind of workflows locally as just scripts, and then it kind of evolved into a product that supports a workflow, and then as it became more consistent through evals, it kind of turned into more of a agentic thing that's being built right now. So, we'll kind of cover how that went.

0:44 A little bit of background on what this does. There's all kinds of different inputs that you might have. You might have a passage that you put as an input. You might have a video, this video that we're going to be doing. You might have a a voice memo, a chat from Discord or company Slack. There's all kinds of information there, and we want to turn it into something that we can share. One of the big things is this is something that you've you may have heard many times is the first part of writing is to have something to say. I think most people that are working on something do have interesting things to say already. And they actually have records of those from Slack conversations, maybe it's the Zoom recording of this. Maybe it's an internal meeting where people are sharing updates or talking about new features or the future of the product. You may have an internal meeting with your company about hey, how do we incorporate these new LLMs or agentic features into our product? That conversation has a lot of context that's probably interesting to people externally to the company. But you don't want to just share the video or share your Slack logs and publish them. And so, that's the first thing. Or if you're coding Claude code or code at agent logs. And so there's a lot of information about interesting things that people have done that can be used to turn into a blog post. And that's kind of the first thing that transforms that makes you be able to use an AI an LLM to write without it being slop because it's actually based on things that you've actually done.

2:19 The second piece There's a couple other pieces that go in here, but the second piece of that this is built on is that writing stuff is personal. AI is not going to know what details are interesting to you or to your audience. It will have an idea because that's what you focused on in your Slack conversations. That's what you focused on in your meeting. That's what you focused on in your agent logs. But it doesn't really know what details are the most interesting. It doesn't know what parts were the hardest for you to do particularly.

2:47 And so because of this, you still need to be heavily involved for good writing. There's a whole lot of transformation style transfer thing taking work that you've done or things you've discussed into writing, but you can't just automate it or it becomes slop. So this whole thing this whole product kind of went through a little bit of a journey. We're going to kind of cover that here. We started with using AI naturally and we ended with moving to more agentic once the results were stable and there was kind of a whole a whole journey in there.

3:19 So the first thing is when I was using it, this was even before I knew it was a product. It just started with vibes. I was just trying to use AI for this. I tried different models. I tried Claude. I tried Gemini. I tried chat GPT. All I was focusing on was content quality, finding small prompts that helped. You can see here. This was kind of an early thing that I posted. Hey, here's a little side-by-side comparison with asking AI to write something and using a different prompt. And so I think it really starts with before jumping into really structured evals or even building it into the product, it was well, can I use something and get some value out of it and get a little bit of benefit? And this was just a lot of iteration, a lot of trying to figure out if this is even worth doing.

4:13 I settled on Gemini. The big reason for Gemini wasn't so much a difference in writing quality between the models. I felt with good prompting, you can get fairly close with all the models, though I did find Gemini was a little bit better at that. But the most important thing was that I found that having a lot of different inputs was pretty important. So, often when I want to write something, just talking things out is easier, creating a screen recording rather than me trying to describe exactly how the workflow works or why it might be useful. I can do a little screen recording, maybe it's 15 seconds or 30 seconds, and so I can say this is the workflow. It might be an hour-long presentation, it might be I built out a stripe thing. I had a blog post about how I refactored my payment system. And so it might be here's the code files it was particularly interesting or I thought it was interesting.

5:10 It might just be a voice transcription from something on Voice Pal. There could be PDFs. For PDF processing stuff, it might be images, whether it's screenshots or pictures of other things. It might be big text files, agent log created through Spec Story. There's all kinds of things in there. And cuz I blog about things as I go quite a bit, I had this post where I talked about here is how Let me get rid of this.

5:37 Here's how I created this post. And so, for this blog post, this kind of gives a little bit of an idea of how I created this. And so, in this case, for one post, I had four screen recordings. They were all very short, 30 seconds to a minute. The actual Python script that was created, a voice transcriptions of my thoughts, why it was interesting, and a PDF of a guide to going viral as a style guide. This was an idea from Hamel.

6:04 And so, the idea was give as much context as possible. And I think that's where a lot of AI usage for writing goes wrong as you say, "Hey, I want to write about the important things that are used in HTMX." Go ahead and write the post. It creates a post, and then it's super generic and slop, and then you say, "Hey, I I I can't use this. This isn't very useful to me." And so, by giving it all this stuff, it already has a lot of context on HTMX, but I've given it what I think is interesting, why I think it's interesting, how I did it, what was I intending to accomplish, what was the actual logic, and that lets AI give you a reasonable draft that actually has my voice. This was an actual post that I ended up writing.

6:49 Here it is. This was kind of one of the early ones. And yeah, it ended up as a real blog post with some of the screen recordings embedded, some of the code snippets embedded, and all that. As I was working on improving the prompt, I was making very small incremental improvements to it, which was helping. The biggest jumps in getting AI to write better quality is when I started studying how to write myself, and that let me be a little bit more specific and use the right language for how to improve writing. I could maybe recognize good writing from bad writing, but I didn't have specific words to describe why some writing was good or bad, and so improving the system required studying that. These are some of the resources that I used on writing well, writing with flare is a course, feedback from friends, so this is Hamel and Danny and Audrey wrote two scoops of Django, provided feedback, Alexis gave a lot of feedback as well, and just studying kind of writing that I liked as well.

7:57 There's some other resources, but these were kind of the main ones. And as I studied that and started to become more of a domain expert myself, that's when the prompts really got better. So, this I think ties back to the course. This would be, I guess, less necessary, so to speak, if I had been partnering with someone who was a professional writer. Because then they could own the prompt, they could do the evals, they could do the error analysis and figure out was this a good result or a bad result, they could do the open coding because they'd be a domain expert. However, I didn't have that, and in order to make a good system and actually know how to evaluate it and know if it's better and why is why it needs to be improved or what specifically needs to be improved, I needed to start becoming the domain expert myself.

8:44 And I was never a terrible writer, but I was never a great writer, either. And that was the purpose here. And yeah, this was probably the crux of it. So, if you're working on something, maybe you can be a domain expert in your field if it's something writing. If you're doing something illegal or medical thing, you probably can't just become a doctor or become a lawyer practicing lawyer to do it, but I think any time you spend improving your own knowledge on the domain, it's really well worth it, even if it doesn't seem necessary.

9:17 And so, there's lots of things that went into the prompt. There were things about nounism, which is this idea that you can often, I can't highlight here, you can often write things in a more passive way, but you can replace those with verbs. There's words that detract from the value. Some users reported you can remove that clutter and just say users reported. If you say users reported, nobody's going to assume that meant 100% every single user said something. The sum doesn't add any value. And so, there's all kinds of things that you can do. Parallel structured lists was something that I'd never thought about before. The second one tends to hit a little bit harder because all three elements of the list are the same lengths. They kind of have a parallel structure. And there's a whole bunch of these that I went through and I got the idea and the ability to be specific about it because I was trying to become a domain expert myself. This wasn't something that I would have thought of or thought of how to communicate it organically without some study.

10:20 And so, as I was doing this, I ended up moving into a product so I could iterate on it daily. It was nice to have the freedom to explore the prompt just in a Claude web or chat GPT, but I also wanted to build it into a product and start building the rest of the user experience around it. There's lots of features that ended up being built. We'll talk about that in a second. In this case, we had a sound that I have a feature where you highlight, it brings up a menu for you to choose some default prompts or type it in and do a diff. And I needed to do something to evaluate this feature.

11:03 As I did evals for that, it followed a similar thing. I just started using it, iterating on it, and that worked pretty well without any kind of structured evals. However, over time, I started having trouble and losing track of things. And the reason I started losing track is because my features and the types of things that I was editing just kind of started blowing up because at first I started with blog posts, but then I wanted Twitter posts and LinkedIn posts, and those were a little different and I was sending out a newsletter and that had a different format.

11:37 I wanted to enforce a particular structure sometimes. If I'm for example for my newsletter I had three sections that I would always include and I wanted that structure enforced in it and that became kind of a new thing that was sometimes in the prompt and sometimes not. When I was writing a post on multi-vector retrieval, we talked a little bit about May spread on the office hours couple hours ago. I was writing a post about how some of the details, the internals of that work and the quantization.

12:09 And I wanted some visuals to explain and make some of the technical concepts around that a little bit more approachable. And so I needed images created. Someone recommend that I hand draw them. I tried. I'm not enough of an artist for that. And so I wanted AI to generate it and so I started building that in and so that I could highlight text, generate an image and have something there. That followed a similar path. I also needed to try to be more of a domain expert. I I read a lot of William Tufte's work. I studied Swiss minimalism. There's a whole bunch of different resources that you can look to be able to be more specific about what you're looking for to make good looking diagrams that don't look super AI generated.

12:54 And those generated images here, I started including YouTube chapters. So for example, if we create turn this into a blog post, we may upload the video directly or we may upload it to YouTube and put in a YouTube URL. And we might want YouTube chapter markers for the sectioning of it. So that was another generation type. And with all these different features, I wouldn't use all of them on every post.

13:24 And so when I did a new post or when I worked on something new, I wouldn't really remember if it was better or easier than the last time that I did things, because I just was there was just too many features to keep track of them all. And that was kind of the the impetus to be a little bit more organized with how I'm evaluating things. I started with open coding, which is really just looking at inputs and outputs, and taking some notes.

13:52 I pulled data from my database, read outputs, annotated things, made it changes, tried it again. I had several different go-to sources that I used. I started out with just doing it through the UI, rather than creating traces, and then eventually I moved on to to to actually having traces as well to automate that. For all of my for all of my data, and my logging, and my tracing, I used Postgres. I didn't use a specific eval tool. I found that I'm I've used SQL for a very long time.

14:27 I'm very comfortable with it, and I found I was just able to put it in there, and not have to add another tool. My main way of doing all this was Jupyter notebooks. What I liked about it is I could query the database directly, and so if I had a tool, and I was missing a piece of data, or I was curious about something, I could kind of explore interactively. I could have viewers. With AI's help, I could iterate quickly. A lot of that is because I do have a little bit of a background with coding, or a lot of background with coding, and I am pretty comfortable with Jupyter notebooks.

15:03 And so, as I was annotating, if I got curious about something, or I got through three examples, and I was, "Wow, it really seems when there's a short bit of instructions, it does better than longer ones." I didn't have to change environments to say, "Let me see if >> >> the prompt lengths Let me sort these by prompt length and then look at the notes and see if there really is a trend there. I could stay inside of Jupiter notebooks and do it the whole time.

15:33 The annotation interface was a Jupiter notebook, but I used AI to create these widgets. I just created a whole bunch of HTML that did it. I didn't carefully review the code for these widgets, but it created them. So, I had here's the user instruction, make it short and concise. This is the text that the user had selected. The full prompt was on the left, the AI response was on the right, and so I can see make it shorter and more concise.

16:00 This was the selected text. Here is the AI response. So, these are kind of the three main ones. And then if something looked odd to me, I could look at the full prompt and everything that's inserted in there, all the writing style instructions in there. But if it looked good, if I looked at the selected text and I felt this was really good writing, then I could move on. At the bottom below, it had these quality, these failure modes that were added eventually, and also notes. And this would save back. Initially, it just saved to a JSON file locally, but then eventually I added that to save it back into my database.

16:36 It's I would just do the annotations in a Jupiter notebook. I needed to add retrieval. I found that with some of these edits, you know, when it selected the text and said made it more con- make it more concise, it wouldn't have all of the source material, whether that was the video or screenshots or voice transcription. It wouldn't have all of it in every single time, because initially I did, but then these edits just took so long because there was so much context in many cases. It would be trying to include a full hour-long video when trying to make a sentence more concise.

17:12 And so that was too slow, and so I removed it so that the prompt would be faster and unfortunately what that meant is sometimes it would make things concise by cutting core details or if I say explain this more, it would make up details and hallucinate. And so added retrieval to give a bit of a middle ground and this was using mixed bread. And so we had edited stuff conflicted with the source. It was very hard to verify things. So even if it was correct, if it added a small detail if you're not familiar with the source material, it was super hard to verify because digging through a raw agent log or jumping over to find where in a video some particular thing was said was really hard to verify. It wasn't so hard when I was writing something that I had just recently done and and I was really familiar with but when I was writing about something I did even a month ago or when I was writing something that somebody else did or worked on that I was less familiar with, it became really hard to be constantly digging through that raw material.

18:23 And retrieval grounded in the source material that it would grab the most relevant chunks, put them in the prompt. This helped quite a bit. And also didn't slow it down too much cuz it wasn't pulling 100% of the source material in just those editing steps. So I did the same thing. I needed to evaluate it and so I had here's the selected text that was being edited. I didn't really need the particular instruction. That's why that's not here. And on the left I had here's the the retrieved chunks and here's the source document of the blog post. And so I had the information in a Jupyter notebook as well and at the bottom I just had a pass fail and some tags and notes that I could I could label in the Jupyter notebook.

19:13 And so all the different operations, whether it was image generation, generating the chapters, the two that we saw, which was text editing, and the retrieval thing, each eval was just its own Jupiter notebook. It had its own setup connecting to the database. It pulled in all the relevant information into Python. It had a section for the actual annotation and the the widgets that are doing the interface. And so, each one just had its own Jupiter notebook.

19:42 And that was how I organized that. Eventually, I needed a data set, a golden data set where it became really burdensome to rerun manually through the UI. So, for example, if I was evaluating retrieval against videos because I changed something, I would often go into my UI, paste in a YouTube URL, let it process that, create a blog post, do an edit, and then I would manually generate these by using the app. I think that's really helpful. It was really helpful to do that for actually quite a long time because as I was doing that, I would find that repetition, I would find things that I found annoying, things that I didn't want to deal with, and just friction in the UX. So, just using my product a lot helped smooth some of those out.

20:37 But at some point, I got really sick of it. And I wanted to be able to manually run stuff. And so, I created a golden data set. These were Once they were annotated, they looked this. I'd have inputs, the instruction, the selected text, the retrieved context, things this. I could use the inputs to generate. There was also in the response output. And so, I could rerun with all the inputs to create a new the response output, or I could change bits of it. And so, I just had every piece of information available for a course that it traces, and I could use this to index in and generate new values.

21:20 >> >> And so, basically it was the core workflow was I'd log all AI operations. I started out trying to be really clever about what I logged, thinking I'd save database space and save lines of code. I found that I I I kept having to add pieces that I thought I didn't need. And I eventually just logged everything, and that worked a lot better. Storage is very cheap anyway. I would browse all the outputs in notebooks. The labeling happened there.

21:50 Eventually, I'd build to golden data set so I could automate rerunning things without doing them in the prompt. And then I would run new prompts, and then I could compare cuz everything was stored back in the database. And so, I could pull any kind of historical information and compare as I went. This actually kind of led the product to evolve quite a bit, and it's going through another evolution now. It started with a single prompt for writing a post. I ended up adding retrieval. There was an automated creation step where it would generate the first draft for you.

22:30 I eventually found that as the editing passes got more and more reliable, I found that I was asking for edits, and it was pretty much always I was always accepting certain edits. I was always doing the same little bit of instruction to edit things, and whenever I would do those, I would almost always accept them. And I couldn't find a way to get that those edits to be created in the initial creation. And I created a process where AI would create the blog post and then it would have two prompts. So, it would kind of do a loop where it would then do a second pass and edit it and another pass and edit it. The idea is that even though it couldn't one-shot them the blog post, it could create the blog post and it could then edit to remove some of the slop. It takes a lot longer for the blog post to be created, but I don't think that was a problem because most of the time I would kick things off and then go do something else and I would come back. And so, that was often my workflow as I'd say, I want to write a blog post about this. I get an idea and I go over, I upload it, I say create the blog posts and then I'm off to something else and then I come back to to to doing the editing later.

23:51 As it got more and more reliable, I found that I could be even more flexible with the workflow step. So, this create and two x edit became an agent skill and everything's becoming agent skills. And breaking it apart and moving from kind of workflow-based, which is how SaaS apps have always worked, to more of a local coding agent working just a cloud code or a code x would, but just with the right skills for writing ended up being helpful. And then each skill would be one job and then they would compose them. For example, this writing style skill is very short. It has a little bit of a description of why the writing style is important, how to use it, where to access it and that would basically it tells it, "Hey, go to this reference file location and read the writing style." And it's directed to use that anytime it's going to write or edit anything.

24:47 The content social is very similar. It's priming it with telling it to read and basically kick off creation of different socials and so, it would go and say, "Oh, let me grab the Twitter prompt." And it would go, read the prompt specifically for Twitter, and then it would generate it. It would generate it by using this Gemini 3 skill, which allows Claude, tells Claude how to make a call to Gemini. Cuz Gemini was the model that does the writing. The reason I did it this way is because I can have it really generic. I can have Codex or Claude code or Gemini CLI or open code. Any of them will work because it's got a skill to call Gemini. It's got a skill to call OpenAI's models. I can have a skill for any model. And so, the coding loop becomes kind of I can use any coding agent. And for places where it really matters, for writing, for Gemini 3 with multimodal capabilities, I can have an agent skill, which can It can send off just the bit that's important to Gemini 3.

25:57 >> >> And then there's other skills. How do you initialize a new project? How do you process an image? Or how do you take a screenshot of a video, which was very important. So, in a lot of cases, such as this talk, if you make a blog post out of this talk, it might be very helpful to have a screenshot of this slide as I'm talking about it into the blog post. And so, you can have a description, but you can also see the screenshot. Or if someone's doing live coding, there might be particular moments where having a screenshot of something they're coding is very important.

26:33 And so, it knows It has a skill I can grab a screenshot when it needs to. And this is the Or you can prompt it. You can say, "Hey, I want a screenshot at 2 minutes and 15 seconds." And it can use the content screenshot, and that gives it the simple bash command that I can grab that screenshot for you. There's all kinds of things. There's a skill for how do you do a transcription of a video to get the text format. How do you grab a video from YouTube? And a lot of these actually use the fact that skills can be composed. For example, the content blog post actually the first thing the content blog post does is it tells it to use the writing skill style skill. And then it tells it to use the Gemini 3 skill to create the blog post.

27:17 And then with the source material, and then it tells it to call it again for the editing pass. And actually before that, it tells it, "Hey, if you've got a link to a YouTube video, but you don't have a transcript of the video, use the content YouTube skill." And just by telling it to do the blog, you can compose the skills together that it needs. And this is the automated creation pipeline. And so we'll see that inside the skills, the content blog skill, there's several scripts that are helpful. And so it first runs it with the core content. It then runs it again with a particular prompt applying some Zinser principles, and it does it again.

27:58 And so it's got this generic these scripts that I can use to call out to the Gemini models. I mentioned this, why two editing passes? These are particular things that are sought after in each of the passes. Those can be eval'd separately. Lots more Zinser rules came into code looking for removing that lots of nouns to use verbs, more more active words if possible, cutting out clutter. In most cases, if somebody says in order to, you can just say to.

28:32 Shortening some of the sentences, getting rid of constructions that don't really mean a whole lot. Another helpful thing for reviewing, even if if you have a blog post or some writing that you're doing, one thing that I find is really helpful, even without any kind of product or tool, is just copy and paste your text into Gemini or Claude or ChatGPT, any of the web interfaces, and say, "Review this as Zen Surfer would." And that's plenty to get a really helpful review.

29:02 So, yeah, I think invoking the name does a lot, but it doesn't do enough, I guess. I kind of mentioned this before, but the reliability is what allowed it to be agentic. So, when you're working with Claude Code, it's first, it's it's non-deterministic. It's has a lot more freedom and flexibility to do a lot more things, and that's super helpful. However, if it's doing stupid things, then you don't want it to be agentic because it's if it's giving it very simple tools where it can do almost everything, if it's always doing stupid stuff or stuff you don't want, or you're having to really watch it and catch it and tell it not to do that, then you almost prefer a hard-coded workflow, because then it's going to work. You're going to have confidence in it.

29:49 But, once it becomes stable and you're "Hey, I have this workflow that I'm following, but now the workflow is kind of tedious because I always ask for the same edits." and then I accept them. At some point, once you have enough stability, you I would rather have the agent have agency and use these things because these tools are kind of reliable enough now that I feel like I don't have to really control every single step.

30:20 But, it took a while to get there, and I started initially by just trying Claude Code and asking it to write stuff, and just plain kind of vibe writing, and I found that it didn't work well enough for me. I just ended up with slop, and I was trying to edit stuff, and it would edit too much. And over time, once I got prompts that were working for me and I figured out the workflow and I figured out I use lemon fox for example to get transcriptions of videos and I found that passing in the transcript along with the video was a lot more helpful than just the video alone and adding in timestamps into the transcript so it's not just a transcript of the text but every few seconds there's a timestamp of where when in the video this thing was said helped a lot so finding out those small details and once I did I started to be comfortable saying okay great now I can just give it to an agent they can use a little bit more discretion but getting that stability first I think is extremely helpful and some things coding that stability already exists you don't need to create something special but for writing I found this that was necessary before it started become useful I can evaluate each skill individually it's very easy to do one calls out to Gemini and so by evaluating it it's really just asking an agent using the cloud SDK to invoke a skill and it just kind of simplified everything this is the interface for it this isn't live yet this is in development hopefully it'll be live soon but the idea is here on the left we've got multiple projects on the right we have our cloud code but it could be code x it could be whatever it's just using that interface it's got all the agent skills loaded in there there's three at the top which are common once you click on one if you click on for this generate blog one all it does is it goes in here and it says slash generate blog with the right stuff and enters it basically what I looked at conductor here what I noticed is yeah projects on the left I did the same thing but I didn't need get work trees for the writing and I didn't use get work trees. I just had different directories for different posts.

32:41 The second thing he's got here is he's got the in the middle panel, that's the coding agent, and I wanted the coding agent. Now, on the right, this had that right half, this is all stuff needed for coding. So, it's Let me look at the file. Let me have a terminal. The setup and the run, you can configure instructions to run it. And so, this panel was things for coding specifically. The terminal is important to have access to.

33:09 And so, I was like, "What would this look for writing?" Well, I still need different projects, cuz I might be working on several posts at the same time. I still need the coding agent, that's where all my agent skills are. I don't need any of that stuff on the right for coding, but I actually, what I need is to be able to edit my posts. And so, I need some sort of markdown editor, and I need something to upload context. And so, basically, what this is it's this interface without any of that coding stuff, the get work trees, or any of the stuff on the right. However, it has a a markdown renderer and editor, so I can really edit the post in there.

33:47 That could put anything I want in it. And here, this is all the files that were created. So, in this case, I've got a blog post, I've got There's a description and a transcript I can look at. It kind of loads it in a nice way. You can see it shows that the screenshot, shows everything here, and I can edit it. And so, this is something that's really important for writing. And then I've got all the social stuff, a carousel thing, or a video long video transcript, and all that stuff.

34:14 And so, heavily inspired by Conductor. And it it really opens things up because it's it used to be where it's oh, well, we need a feature to do this particular thing, and now developers would need to build it, and now it's oh, you need to replace links or do this special thing, or you want to bold every time the product name comes up or italicize it or something. I may not have a feature built into this, but you can just ask the coding agent to do it. Now, if everybody is doing the same thing, I would want to create a skill or build some feature in to do it, but they don't have to wait for me to build a feature. They can just do it.

34:53 In terms of having the users be able to modify their own interface with the coding agent and things that, I'm not sure. I do want to experiment with it. However, the more flexibility they have to change the core application, the less, I guess, guarantees I can make. For example, if I say, "I don't want people to break their own thing." And how important it is it? For example, if I If one of the things I want to say is, "Yeah, you can rely on writing posts here and them not get leaked." Which if If you're just blogging for fun, not a big deal, but if you're someone, I don't know, Accenture or an OpenAI or some big company, where people might be trying to find It's just It's hard for me to make guarantees about security, about how well it works, about the workflow, if I'm also making guarantees that they can change all of it. If they can change it, I can't guarantee it's going to work.

35:53 So, I'm not sure. I would love to do it, but I'm nervous. I mentioned this before, but skills compose. And for example, when to write a blog post and social content, and you put the video in, the transcription thing runs, the screenshot thing runs, the blog post runs, and here it's also calling the writing style prompt skill. It's doing the editing passes. This blog is actually calling the Gemini 3 skill. It's calling the content social skill. And so, it can really combine these in any way it wants. Oh, here's the content blog.

36:31 So skills also often use sub skills. So the content blog basically says, "Hey, if there's video but no transcript, call the content transcribe thing. Before writing, use the content writing style skill. When you want to create a to create the blog post, call this skill. Do the editing pass of this skill." And some of these skills are kind of higher-level skills tell it to to call other skills. have that information. And so they really do compose quite well. I'm using Claude code, but the nice thing about these skills is that the writing calls the skill for Gemini even though it's a Claude code thing. So the nice thing about this approach too is that if Anthropic decides, "No, you can't do this." and they turn it off, it's I don't fine, I'll just change to open code or change There's nothing weird about that. Yeah.

37:19 I'm probably going to change away from Claude code for the side panel and switch to something else because cuz Anthropic's been a little aggressive about that, but yeah, I think it's hard to say what the future will hold there. Bring your own key, there's a little settings panel so people to paste in their keys if they want. So what I'm currently thinking is this can be a this editor can be a electron app, which it is now, local electron app. VS code is an electron app, for example, so it's pretty flexible.

37:51 So my current thought is that's it's that and then all the collaboration features would be a web app. And so I would use the local electron app to do all the powerful editing. I'd have it locally, which is nice because it's the most flexible. So all the files will be downloaded locally. If I'm hey, I want to write my own script, all the files are there. It's a tool that should do everything you need, but if you feel you want to ask Ampere to do something else or run your own workflow locally, you can do anything you want with it. And so, I'm thinking that local app will be kind of the main writing tool.

38:29 And then, the cloud will be all the team and the collaboration thing. So, I would sync a post and send a private link to to you, maybe to edit it. Or, I could set up a team where all of my maybe I have three or four people who have access to look at any of the posts that I've written, even if they're not public. And then, build review features, and so people can review and add comments, and that can be brought back into the app with the context for the models, if you want.

39:00 And so, currently I'm thinking local app for all the editing, everything I kind of showed, and then there'll be a web app for team and enterprise and collaboration workflows. Deployment, publishing, and all that. Okay. And just so people know, this is the old one that Isaac, I'm sharing my screen, that Isaac has has built. And then, you can go to Isaac's blog. isaacflath.com I'll show here. And this is his blog. And you can see you can keep up with what he's doing. He has post recent blog posts. He's wrote a draft, but yeah, you can keep up with this. On Monday, I'm starting I'm going to be working with Spec Story. I start working with them as a founding engineer. So, a lot of my focus is going to be on helping companies and teams code with AI better. So, there's a lot of things that are helpful about coding agents, but it's still a little bit difficult for big teams of people to all be collaborating and building the same product together.

40:01 And they've got some stuff that they're working on for that. If you're interested in that, or your company's interested in that, let me know. So, that'll be the focus. And part of that is I have a draft of a book I wrote about different ways, patterns to use coding agents. I'm working on figuring out if I should make that a book or maybe I'll turn that into a little mini free course or something that I put on my site or something that. The important thing for using a coding agent or really any LM product or anything that is that you give it all the context. And so this context can come in a lot of places. It actually has quite a lot to similarities with write a draft if I'd be an honest is that you need the context in order to build anything, to write something but also to build a product and to build yeah, to do coding. And so some of that context comes from the coding agent. I may open up cloud code or amp or codex and be researching different dependencies and libraries and brainstorming ways to do it. All of that is useful for actually building the final product. It can save coding agent history. In the same thing, a lot of internal meetings where if you and I got on a call and we were trying to think about, okay, how can we make a better notebook interface or how can we create an eval product or how can we do that? The discussion between you and I would be really helpful context. And specifically and people record these discussions all the time. What you really want, I think, is a recording of this and a log of all of the decisions that were made or not made. Okay, we decided we are going to build something to help with evals.

41:38 Okay, what were all the points and the things we considered related to that decision? We decided that we wanted it based on Next.js. Well, what were all the things we considered? What were all the alternatives we considered? And why did we choose Next.js? That might be useful context if we want to explore more or we want to give it there's probably a lot that went into that decision that's helpful information about our intent and how we built things.

42:06 And so being able to group all of this stuff coding agents, conversations, whether it's a video call, Slack things. And as new discussions come up, being able to group these under the kind of decisions that you're making, whether it's big or small, maybe it's a pricing thing, maybe it's a UI thing, maybe it's a a very small feature, I think gives it the right structure to give both human and AI context to kind of be aligned on what are we doing, which branches are we taking, which branches are we not taking cuz we might want to review those in the future. But yeah, thanks for coming.