Section Insights
Opus 5.5 Enhancements
What improvements does Opus 5.5 offer over its predecessor?
Opus 5.5 aims to communicate more naturally and reduce 'slop' in writing. It prioritizes important information and adheres to user-defined writing styles, making it easier to follow long sessions.
- Opus 5.5 addresses common feedback about writing quality.
- It enhances natural communication while following user instructions.
- The model aims to reduce unnecessary complexity in writing.
Challenges with AI Writing
What issues arise from AI-generated content?
The AI sometimes misinterprets prompts, leading to defensive or overly complex writing. The accuracy of the AI's output can be inconsistent, and it may not effectively address the intended audience.
- AI can misinterpret user intent, resulting in inappropriate responses.
- Complexity in AI-generated sentences can hinder comprehension.
- The effectiveness of AI writing is still under scrutiny.
Testing AI Responses
How can we evaluate the quality of AI-generated responses?
By intentionally generating 'slop' and comparing responses, users can assess the AI's ability to produce coherent and engaging content. The goal is to determine if the AI can craft meaningful replies.
- Testing AI responses helps identify strengths and weaknesses.
- Generating slop can reveal the AI's limitations in content creation.
- Comparison of outputs can guide improvements in AI writing.
Engagement in Social Media Responses
What makes a social media reply effective?
An effective reply should be short, thoughtful, and engaging, prompting further interaction from the original poster. The AI's ability to craft such replies is being tested.
- Short and thoughtful replies are key to social media engagement.
- The AI's performance in generating engaging content is being evaluated.
- Both versions of the AI struggle with producing high-quality responses.
Evaluating AI Improvements
What changes were made in Opus 5.5 compared to Opus 5?
While there are some improvements in Opus 5.5, the overall structure and output remain quite similar to Opus 5. The enhancements may not be substantial enough to declare a definitive superiority.
- Opus 5.5 shows minor improvements but retains similarities to Opus 5.
- The evaluation of AI outputs reveals a need for further refinement.
- Both versions require significant editing to achieve desired quality.
Transcript
0:00 Opus 5.5 communicates more naturally, addressing some of the most common feedback we've heard. Like basically, slop. Like, okay, it puts the most important information up front and follows the writing rules you give it, makes long sessions easier to follow. Okay, so let's see. Let's see this like image. >> So, it's like two parts of that, right? There's like does it naturally write well and then communicate naturally and then the second part it follows instructions so you can have it right in your own style. Like this would be the two things they're claiming here, right?
0:32 >> Yes. Maybe you need like a prompt to reduce slop saying also, okay, like maybe it'll just write less slop. Okay, so what I did like last time we did a writing thing, I vibe coded this like app that does side-by-side review. I'm sure there's like different way to do this, but this is like an easy way. So, I'm comparing like 5.5 to 5. And what I basically what I'm doing is saying like, okay, I'm going to ask this question, can you describe write a short article like a FAQ that tells me or that describes what AI eval's are. You have Opus Opus 5 vs. 5.5. Okay, let's see what happens. It's exploring the data. It's exploring like the context, you know? Cuz basically you're having it write an article based on some context, which is kind of like what you do if you're trying to write something. Okay, you got Opus that start >> It looked like a lot Opus 5.5 did a lot fewer tool calls.
1:29 >> Yes. >> I don't know if that's good or bad, but I wonder if it read the same amount of context. That's another thing, I guess. >> Let it generate a little bit and then we can do a side-by-side. We could see which one do you think is better. It took some liberties here. Like what is eval in one sentence? It's hilarious that it says what is eval in one sentence and it gives us an entire book on it.
1:50 >> Well, the sentence I mean if you use an if dash instead of a period, the sentence is it over, right? I I so starting at first, I do Opus 5.5 added that extra a short FAQ. I would probably delete the word short. >> Mhm. >> I'm like, is that that doesn't seem people know what an FAQ is. So. >> Yes. Based on course reader, blah, blah, okay. What is AI eval? So, this is where the meat is. An eval is a way to measure your AI application based the way you need it to. It usually isn't one test or one score. It's a process.
2:20 Okay. So, I'm just going to record the vote. And like, let's try something else. Like, let's let's do something completely different. Like, okay, let's give it a writing task. Maybe let's give it a writing task based on your blog posts. Okay, let's go to your writing six things I tried with Jev. Okay, cool. >> Sure. >> Let's try that. Write a summary of this post. >> For >> Make sure it's Okay, yeah. For >> people who are skeptical about Jev because classifiers have existed before.
2:49 >> Okay. I'm going to say be have a warm, professional tone. And I'll even like go further. I'll say like, use this plain writing skill. So, I'll say like, use this plain writing skill from our friend Shreya. I'll just say like, okay. I'll just tell it to look at this markdown file as guidance. So, look at this Wait, let me tell Codex to like, let's make it blind. Make the model names blind. And then allow us to toggle them when we want to reveal. Okay, let's jump back here.
3:25 Okay, Opus 5 is trying to do stuff. 5.5 is still going. Feel like it got stuck or something. Six things Isaac tried with Jev summarize for skeptics. Little bit too literal of like the title. >> Yeah. >> Okay, Isaac flat tested type save dot Jev or type save Jev, a judgment model on six jobs from his own work. And Jev scored about the same as Gemini 3.5 flash on accuracy on answering four 14 times faster. If your objection is that classifiers have been around for years, the post doesn't argue against you. Oh my god. Okay, like this really bad writing like whose objection are we talking about? Like whose like you know, it's like it's it's taking my it's like a over steering based on what I told it and it's like assuming that like I'm writing about someone that has an objection.
4:18 >> Yeah, it's doing that which could be we set it up as fine, but it's like it comes off as like I'm immediately responding to somebody who's like super defensive, you know, like they're really defensive and this is there. Yeah. >> Yeah, this is the kind of stuff that like drives me crazy. Okay, the accuracy the post doesn't argue against you. It's like okay, this feels like AI writing. The accuracy numbers are a tie or small loss in most of the test. So the case for Jev's rest on response time and not having to build a classifier first. Wow, that sentence is so insanely complex. I have to like concentrate really hard to like get that get there.
4:58 It's a lot of it's a lot of cognitive work for something that doesn't need to be so complicated. >> Could be yeah, could be super simple. >> Okay, summary of six things I tried with Jev. Not much better title. If you have built text classifiers before, you'll recognize most of the tests in Isaac class post. So it's reasonable to ask what Jev adds. Amazing, that's so much better. >> Better. >> Yeah, the post answers with results from Isaac's own work. He used one model type safe judgement model Jev for six different decisions about AI and documents. For each decision he asked Jev a plain question and the post doesn't mention collecting train data or training separate model. Okay, it's a little bit of embellishment, but okay. I mean I would if he already knows winning. I mean I feel like B.
5:44 >> Yeah, I feel like that sentence like the point that it made there is could be helpful. It would come later. It would come like I'd show the examples and be like now you didn't talk about how he created the data set and so there's like it would be like a caveat thing. But it started out strong and then it does this and then the sentence right after it kind of jumped into like listing facts. >> It gets really bad like right here. Like all this needs to get deleted. It's going into a rabbit hole really fast.
6:10 >> Yeah, it doesn't need to mention in this earlier mention.35 to.52 seconds. Close accuracy to Gemini flash might be okay, but like that's getting too specific for the intro when we haven't even like that those numbers don't mean anything cuz we don't even know what the task is. >> Yeah. The model trained on label examples to answer one fixed question. Okay, yeah, that's true. It's traditional is the keyword. >> The weather an email is spam. I don't think I had that in the post. I think it made up that example. I don't love it cuz it's kind of a cliche, but I'm not terrible.
6:40 >> A new example need a new question answer usually >> Label new examples. Isaac's six tasks each take a different question. >> that's a weird example. >> That's not good. That's not good. E.g. whether a passage I mean how in the hell I'm like how can you even like imagine writing a sentence like this? Like >> It's it's trying to do like a summary like overview of like here's the tasks he tried and they could do that if you were setting it up as like a table of contents to let people do, but then like each of those examples would just be bullets rather than >> Yeah, it doesn't make any sense.
7:18 Actually it doesn't like did it even say >> Like tasks don't ask a question. Like my tasks that's the issue is it didn't say it's confusing cuz it's like Isaac's six tasks each ask a different like the task doesn't ask the question. Like like I asked the question. >> We don't even know what these six tasks are. Like where did this like when I read this I'm like where what six tasks? Like there's six tasks? And this happens with AI a lot. Yeah, yeah, yeah. So like, but I don't know like six things?
7:44 Is that six tasks? Like what is going on? And this is the kind of thing like AI like loves to just push information in places where it shouldn't go or it doesn't flow. So okay, I would say like B or like 5.5 wins, but it's still slop. >> Yeah. Slop will be a problem for maybe a year or two, only as long as weak, less capable slop producing models are still being used. I think we're seeing that five five is still it seems better so far with our sample size of two.
8:15 But it seems like it's still slop. >> Okay, so let's try to produce slop on purpose. >> Let's try and make a good Twitter response, like a good Twitter reply, and see if it can do it or if it just comes out slop. >> Okay, let's actually read the post so we can like guide it, okay? RLM is the most important function of my pie harness. I'll go skim it. It's the most most important foundation of my pie harness, that's interesting. It's seeded with late interaction retrieval results. Then you have a setup, you have iteration loop. Why does this work? Okay, you have a richer shell, keeps main agent context clean, stacks good ideas. Let me riff on this for a second, okay? So, understand the post in detail, also understand the link from the post of the full blog post. And think about a thoughtful reply that asks him about how does he know that it's better than other approaches like without RLM's or other coding harnesses? What convinced him that this is the best approach? Please draft a reply. Assume that we are familiar with coding agents in general. And we want to understand sort of his perspective. And I can come over here, have a new session, I can copy and paste this.
9:29 And I can say like draft, okay, like a thoughtful reply in the style of a LinkedIn influencer. What was this thing? There was like try alternate mirrors. Okay, I see. It was trying to pull the tweet. Did we ask it to write a Twitter reply? >> I did. I think it's like it gives give me this preamble of like what it did and then it's getting into the reply. Great write-up. A plain sub agent with RG This sounds exactly like a bot. Great write-up. A plain sub agent with RG Your pilot search also keeps making this clean.
9:59 >> Say what? Talk like that? Does that sound human? Like in what universe does that sound human? I get so >> Yeah. >> upset like when I see like that, I'm like I just I'm like let me block this person right now. It's like insane. Like that is there's nothing about that this thing right here even though we put a lot of effort into making it good, it is really bad. >> There's a whole bunch of like sources that I I look at and I have an LM judge for like to stay on top of document stuff. It's really funny because I start to learn which places use AI for their posts and I just dread reading them and so as a result like I just don't want to read through it cuz I know there's even a 75% chance that's like kind of slop.
10:42 So I don't and then I don't make content about them as much as I otherwise would. >> We are just comparing models. So 5.5 versus 5, we're seeing like does 5.5 produce less slop? Allegedly it's supposed to produce less slop. Okay, on the right Opus 5.5. Okay, this is like definitively slop and I would say this one great post making the traces live notebooks is really nice touch. >> Okay, I think it's better than the other. Let's look at the other.
11:06 >> Okay, you think it's better than the other? Okay. >> The notebook is live experiment is the part I haven't seen elsewhere. What I can't tell from the post is the counterfactual. Three ideas stack. Yes, I think 5.5 is better. It still is slop. It's still easy to identify slop, but maybe for the next time we come in we need both failure button. >> Okay, this is the one where we try to write like a LinkedIn influencer. So we were trying to write like slop on purpose. So let's see. You know, according to their Twitter post, it may be 5.5 will write like even more cringe.
11:36 I mean, it's definitely Okay, like which one do you want to read? >> This one is the I'm an influencer prompt that we're about to look at. Okay, most teams are still arguing about chunk size. And then let that sink in with a point down. I'm reading the left. So, that's that's >> No, but it's like super interesting, right? Cuz like 5.5 did the same pattern right here. It has a hook and it has a read some with an emoji here.
11:59 >> Let's try one more version of Twitter of social media reply where we take one, give it something Oh, and five has the it's not X, it's Y. >> Okay, let's let's take one of Brian since Brian is such a good sport. Maybe just this one here. It's a short. You know something is good when it is simultaneously used to argue conflicting viewpoints. Okay, so you want to like write something in response to this? >> Yeah, so let's do this and let's have the prompt be something like general.
12:27 Like craft a reply that would be interesting to the person who wrote the post and would likely get them to engage. It should be, you know, short and thoughtful to create meaningful engagement. >> So, this is pretty short. Now, I will say like 5.5 is not shorter. Okay, so let's go back here. Okay, so we got five on the left, 5.5 on the right. Okay, so which one >> Start with the left, I guess. Okay, horoscopes clear that bar, too. The tell is whether both sides are quoting the same line or just the vibe. I feel like But we got to pick the better one. I guess I get the horoscope point. The horoscope kind of can mean anything. You can interpret it any way. The second sentence is super bad. Okay, or it's a test my tiebreaker. Both sides painting >> I don't know, man. It's like >> This is Yeah, this is really bad. So, I think we do not have to worry about engagement bot that is not detectable.
13:21 >> Yeah, I don't think we have to worry like 5.5 has killed slop. They both failed to me. I don't which one is better, honestly. So, I'm going to say they both failed. >> Title and abstract for conference proposal. That's a good one. That's a really good one. >> Yeah, okay. Yeah, I'm just pointing it out my eval's material. Just cuz I'm trying to make it realistic, right? Like when you write something, you're like you're trying to point out some context you have. And I'm saying like, "Okay, write a title and abstract for a conference proposal conference proposal for a talk." It's kind of a weirded weird sentence, but okay. Are we ready to go?
13:50 >> I mean, that is as good context I mean, you got a whole book. So, that's as good context as you could hope to give for something like this. >> They're both running on high effort default cloud code prompt. And then we'll see. We'll see like if one your LLM judge is a classifier, start measuring like one. It's not bad. Although, I wouldn't try like it's too it's too hypey to me. >> Feel like it could say, "Measure your LLM judge like a classifier." And that would be better.
14:18 >> Now, their post says says that 5.5 is always shorter. I don't know if I feel >> Okay, so the bottom the left one had a sort had a attendees leave with, which is on a 5.5. >> This one has attendees leave with two, but it's a lot shorter. >> A checklist for building a judge and a way to see the judge they are run versus a criterion for choosing code checks over >> No, I already don't like this. It's too much. It's too much. Like I don't I like your LLM judge is a classifier, start measuring like one. Okay, give me some alternates. Interesting.
14:52 >> That title actually somehow is better with 5 to 5.5. 5.5 is your LLM judge is a model, validate it like one. I mean, they're both kind of bad, but at least the left tells you >> Yeah, let's do the abstract. Okay, so five. Most teams write an LLM judge prompt, eyeball a few outputs, and start reporting the number. That number is unfalsifiable. Okay, I would never write like this. This is very confusing. The judge itself is a classifier with two independent error rates, and nobody has measured either one.
15:23 >> Yeah, but you haven't told us what either one is either. >> Yeah, it's just so I'm like when I read this, there's so many problems with this. Like I don't even This is so bad. This is like classic slop. There's nothing good about this. >> This talk walks through the full pipeline for building a judge you can defend using customer support agent as the running jumbo. >> Let's read Opus 5.5. More and more teams measure quality with LLM judge. Colon, a prompt asks a model whether a response is good and the answer goes on a dashboard.
15:52 >> Okay, it's trying to like be too dramatic. It's trying to >> It's not It's a little bit better than the other one for sure, but it still sucks. Too often nobody checks that judge against human labels. >> This talk shows how to build a judge whose numbers you can defend. The running example is a customer support agent that tells shopper your refund is complete. Wow, okay, it's very parallel in its structure. Like it's like really interesting that like wants to say the same thing and organize it the same way.
16:20 >> So yeah, I wonder what they did. Did they change the data that it was trained on? Cuz it doesn't seem like significantly cuz it's still mirroring the same thing. Was this just training? What did they do to 5.5? Like what did they do differently to try and get it to write different cuz it's still so similar. It doesn't seem like that. They made some improvements it seems like, but it doesn't seem like they made like a complete like major change because of how similar they are.
16:43 >> They're extremely similar. Like the title is similar. The first sentence is fairly similar and then the structure is like we start before a prompt. You start for a prompt. Okay, we looked at how many examples? Let's see. Like we looked at So we have like three wins. >> We have three wins, one where both failed. >> I'm going to say I don't like either. I mean they're both like just you have to edit the hell out of both of them.
17:06 >> There is no slop in 5.5 still that we would need to do a lot of editing. >> Yeah, I mean like we have three, but then we did six. I think like we passed three. So it's like very unconclusive. I don't think I we can say like five is five is like definitively better. It may be it was like a little bit better in some of them, but I don't think >> Can we go back and review the first one we did and see if do we hate it more now that we've seen a few more examples or do we still feel like like will we fail them both if we read it again?
17:32 >> Yeah, I mean dude A is definitely left hand side definitely lower. It's clear on this one. >> The basic unit is the evaluator. The reader defines it as a procedure that maps a trace and relevant context to a judgement about a specific property. A trace is a full record of one request. >> It starts okay, but then like yeah, as you go down it starts to you get bad, right? Like I'm sure like if I kept reading this I would start hating it too eventually. But I'm saying like okay, just from the intro I feel much better with 5.5 in this particular example.
18:01 >> I guess I feel like in all of them I thought 5.5 was better than the other, but I didn't I didn't think any of them were great. >> Okay. You thought even for this one, this conference talk one? >> Yeah, I mean I liked it better. I guess when I look at this, my thought would probably like in both cases I would probably start over. So, like we start before writing any prompt. >> Slop is not dead. 5.5 might be a little bit better in some cases, but it's not like a landslide by any means.
18:28 So, I think slop is still here. Like it's very much here. Like it has not been solved. >> For the ones they reviewed, the conference talk this person reviewed, the titles were em dash written and the abstracts were mostly generic. Like they'd asked an LLM for ideas on topic X and all gotten the same responses back. I ended up prioritizing abstracts that had specific enough examples to make me think they'd written it themselves. >> Fascinating. Wow. So, like all the abstracts were the same.
18:55 >> Yeah, I mean we reviewed six and it already feels like painful. >> All right. Thanks. All right. See you. >> Real quick. If you're building an AI product and you're stuck on how to improve, we put together this free resource on evals to help you. It draws from more than a thousand hours of working with companies on building AI products. And it's customized to what stage you are at. So, this is how you can use the guide. Go to bit.ly/evals-guide.
19:21 When you open the guide, it'll look like this. And what you do is you select the stage that you're at. Maybe you're new to evals, you don't even know what evals are. Maybe you do have evals, but you don't really know what to test. Or maybe you don't trust your eval scores. Or maybe you think your product is just too hard to eval. Or it's too time-consuming or expensive. Whatever the case may be, we have a detailed guide customized for what stage you're at. We've worked on it for over a year and are giving it away for free. Go to bit.ly/evals-guide and we'll send it to you.
Summary
- Opus 5.5 is designed to present important information upfront and reduce unnecessary complexity in writing.
- The comparison between Opus 5 and 5.5 reveals that 5.5 generally produces less "slop," but both models still exhibit issues with coherence and clarity.
- Users noted that while 5.5 improved on certain aspects, it still struggles with overly complex sentences and irrelevant details.
- The models were tested with various writing tasks, including summarizing blog posts and generating conference proposals, with mixed results in quality.
- Both models failed to consistently produce engaging and human-like responses, particularly in social media contexts.
- The conversation highlights the ongoing challenge of AI-generated content maintaining a natural flow and avoiding redundancy.
- Despite improvements, "slop" remains a significant concern, indicating that further advancements are needed in AI writing capabilities.
- A free resource on AI evaluations is offered to help users improve their AI products, tailored to different stages of familiarity with evals.
Questions Answered
What improvements does Opus 5.5 offer over its predecessor?
Opus 5.5 aims to communicate more naturally and reduce 'slop' in writing. It prioritizes important information and adheres to user-defined writing styles, making it easier to follow long sessions.
What issues arise from AI-generated content?
The AI sometimes misinterprets prompts, leading to defensive or overly complex writing. The accuracy of the AI's output can be inconsistent, and it may not effectively address the intended audience.
How can we evaluate the quality of AI-generated responses?
By intentionally generating 'slop' and comparing responses, users can assess the AI's ability to produce coherent and engaging content. The goal is to determine if the AI can craft meaningful replies.
What makes a social media reply effective?
An effective reply should be short, thoughtful, and engaging, prompting further interaction from the original poster. The AI's ability to craft such replies is being tested.
What changes were made in Opus 5.5 compared to Opus 5?
While there are some improvements in Opus 5.5, the overall structure and output remain quite similar to Opus 5. The enhancements may not be substantial enough to declare a definitive superiority.