transcribe

Astra 6.2 Is Unusable

Theo - t3․gg · 45m · transcribed 2d ago
More from Theo - t3․gg Business
𝕏 Share ▶ YouTube 📥 PDF 🤖 .md

Section Insights

# 0:00

Introduction to OpenAI's New Models

What are the key features of OpenAI's 6.2 Astra and 6.2 Sol models?

OpenAI has released 6.2 Astra and 6.2 Sol, which are seen as significant competitors in the AI code space, especially against Anthropic's recent models. Astra shows superior performance at a much lower cost compared to previous models like Fable.

  • 6.2 Astra and 6.2 Sol are major releases from OpenAI.
  • Astra outperforms competitors while being significantly cheaper.
  • The models are released at a crucial time in the AI landscape.
# 9:09

Model Comprehension Issues

What problems does the model face in understanding user prompts?

The model struggles to distinguish between different types of knowledge and often generates irrelevant or confusing questions, indicating a lack of comprehension.

  • The model fails to understand context and user knowledge.
  • It generates nonsensical questions that hinder user interaction.
  • There are significant issues with the model's ability to process prompts effectively.
# 18:18

Pricing and Availability Updates

How have the pricing and availability of the new models changed?

6.2 Sol maintains the same pricing as its predecessor, while 6.2 Astra has seen a price reduction, making it more accessible for users.

  • 6.2 Astra's price has decreased to $8 per mill.
  • The cash input cost has been significantly reduced.
  • Overall pricing changes are favorable for users.
# 27:27

Critique of Model's Text Generation

How does the model perform in generating text that matches the tone of existing games?

The model fails to replicate the writing style and tone of the game 'Outer Wilds,' leading to disappointing results that do not align with the original content.

  • The model struggles to match the tone and quality of established game writing.
  • Generated text is often seen as offensive by fans of the original game.
  • There is a significant gap in the model's ability to produce meaningful dialogue.
# 36:36

Challenges with Long Tasks and Knowledge Gaps

What issues arise when the model is tasked with long-term projects?

The model exhibits a lack of understanding of knowledge gaps, often asking irrelevant questions that block progress on long tasks, making it difficult to achieve desired outcomes.

  • The model's inability to recognize knowledge gaps complicates task completion.
  • It frequently interrupts workflows with irrelevant queries.
  • Users may need to implement additional systems to manage the model's limitations.

Transcript

0:00 Oh boy, they're finally here. OpenAI has introduced 6.2 Astra and 6.2 Sol, two of the most hyped model releases of a whole year. These models are coming at a particularly pressing time because Anthropic's been dominating in the AI code space for a few months straight now. Ever since Opus 5.5 dropped, it feels like they can't be touched. Anthropic has come out with some new models that really show some incredible capabilities, especially when you consider the price. On artificial analysis, these models are showing performance comparable to Fable on everything other than Max, which as I've said before, you should ignore Max, all while at much cheaper prices. When you look at the linear version, it shows just how much cheaper these are compared to something like Fable 5.2. In my own personal benches, 6.2 Astra shows dominance across the board and 6.2 Sol is a very close second place at a way cheaper price. Astra cost about half as much to run these benches as Fable did and Sol was almost as low as an eighth.

0:51 Truly incredible when you see all of these things, right? So is this a home run? Did they knock it out of the park? Is OpenAI back? This is a tough video to make because all of the tools and techniques I have built to measure a model, this one shows great numbers. By all accounts and all measurements, this should be one of the best releases of the year. And that's why it really hurts me to say I I hate using this model. There's a lot of layers to this one and there's a lot of good too, but I was hoping for a lot more. This model's a meaningful bump in capabilities. If you want to see just how hard of a math problem AI can solve or just how beautiful of a 3D environment a model can make, 6.2 Astra is neck and neck with the best, if not ahead of it. But that's not all there is to models nowadays. There's a lot of layers to how it feels to actually work with these things and that's where I have the most issues with this release.

1:48 I wanted to love this model, but I don't. And while it has gotten better the last few days of testing, it's still not quite where I want it to be in order to start moving things off of Claude. It's a lot of layers to this one and I'm probably going to get some trouble for the things I'm about to say, including a letter that I wrote to OpenAI during the early testing that was meant to vent my frustrations. I'll share all of this, including the letter and more, after a quick word for today's sponsor. There's a strange and kind of cruel irony around frontier models right now. On one hand, they're more than happy to write lots of code, much of which is not secure. But on the other hand, when you ask them to help secure your code, they freak out and say, sorry, you're not allowed to do that, and then don't actually make sure your code is safe. So how are we going to protect our code from all the crazy exploits that we're all seeing everywhere right now? Today's sponsor is Coderabbit, and Coderabbit Secure is one of the best places to get started here.

2:34 There are three dimensions of security that we should be thinking about here. There is proactive security where every single time changes are coming in, you're tracking and making sure the code is as likely to be safe as reasonably possible. There is the trends of security over time, which is how well is your codebase staying secured, how quickly are exploits being patched and things like that. And then there is the depth of the code that already exists.

2:54 Are you auditing what has been done in the past with these new tools and combing through, finding anything that might be hiding deep in the codebase? Coderabbit's here to help you address all three. Secure's code review is meticulous, hunting for all of the places things can go wrong. But more importantly, deep scan will do an audit of your existing code that hasn't been checked before to find things through its mapping of where data goes and what can be accessed how. When I got into software, it took years for exploits to be taken advantage of. When issues were disclosed, it would take as much as 847 days for anyone to bother trying them.

3:25 Now it happens same day. When an exploit comes out, everyone's trying to do it. The only way to prevent this is to prevent the issues from getting in the codebase in the first place and ideally fixing them before they're even discovered. Secure your code today with a 14-day free trial at soidev.link/coderabbit. In case you couldn't tell, I'm a bit under the weather right now. I just got back from a Twitch Con trip with my team and got pretty sick when I was there.

3:49 And now I am coming off of a pretty strong fever. It is almost It is exactly now 11:00 p.m. as I'm filming, and I just spent a bunch of time re-running my tests because there were changes made right before release to the model that I had not had a chance to experiment with. Despite all that, did my best to pull it together to come film this video. Hope you guys can understand why I'm a little uneasy. Let's start with what OpenAI had to say about these models. I'll also be upfront now. I have spent much less time with Soul than I have with Astra in this release cycle. If you guys want a more focused dedicated Soul video, let me know in the comments and I will do more tests there in order to get a good feel for it. I'll talk a bit about how people are using the two models, in particular OpenAI employees, in just a bit. But for now, Astra's our focus. We're introducing 6 2 Astra, our new frontier model for your hardest projects, and 6 2 Soul, our efficient and versatile model for everyday work. Both models deliver better results on demanding tasks with fewer output tokens so you can get more work done within your usage limits. For business and developers, this means frontier performance at costs that make sense at scale. We shall see. 6 2 Astra sets a new quality bar for AI in complex professional work. Its slides, spreadsheets, and documents read less like AI and more closely follow the templates that you provide in the conventions of your field. It writes cleaner code and builds front ends that look and work better, particularly for websites, gaming, and 3D design. I normally would read through all this before I start refuting points, but I I need to start doing it now because otherwise I will go insane and I'll feel dishonest. And I know not all of you watch to the end. By the way, if you can, prove me wrong on that. Watch all the way to the end of the video and hit that red button to subscribe, because I don't think it's red anymore. I don't care. Hit the button if you haven't yet because we put a lot of work into these videos and it helps the algorithm and our team know that you're liking what you're seeing. Also, try and real hard to break that 600k line by the end of the year. Anyways, let's start refuting things because I I do not like how they frame this one. In particular, saying that the things it puts out read less like AI, maybe, but they read more like somebody who doesn't speak English.

5:43 Here's where I have to reveal something that I'm not necessarily proud of. I've lied to you guys in a handful of recent videos. Remember in the recent videos I did about the new Fable as well as the new Gemini models where I showed an example from Terra about how it was really bad at summarizing why my machine was slow and why my work trees were taking too long to be created. If not, you don't need to see the original video. Just know that I lied when I said it was Tara. That was a disguise I had in my editor, D3 Code, in order to hide the fact that I was early access testing the new Astro model. So that absolute slop I showed you guys where it talked about all these weird internals with PNPM and the method I was using with XFS and VDO without mentioning the fact that the disk itself was just slower. It like briefly touches on it here, but it's not any of the parts it highlights. That was not an example of a small model being bad. That was an example of a very big, very verbose model not knowing how to prioritize information correctly. This is going to be a recurring theme because this happens a lot with this model. Just to give a handful of examples that were particularly frustrating to me, I gave the model access to one of my Macs in order to have it try to create its own browser and the absolute nightmare situation it created for itself is still horrifying to me to this day. I'll show you the final screenshot in a sec, but I want to show you a bit of how it got here. One of the things it asked me, and I know the formatting in this message in D3 Code isn't clear, the first half is what it asked me, the bottom half is what I responded. Randomly, I just had this thread that was running the long goal of creating this browser, say, when can Pro Mini be quiet for the first command T test? The latest sample averaged 0.18% idle. The test needs 50%.

7:24 I can keep working on the browser while waiting. I had no idea what was going on, why this machine had no resources available. It didn't give me any indication of what might be going on. So I said, what is even using this machine? Is this a bunch of iOS Sims? Clear it out. It's yours to use. To which it responds, can you unlock the disposable QA guest? On Pro Mini, open the already running Tart window titled Her whatever, unlock the manage via Tart account, and reply when it's normal desktop is visible. The automated console shows only wallpaper. No password needs to be sent here. I don't even know what Tart is. I had nothing to do with any of this. It just starts saying things that are within its own internal reasoning and presumes that you know everything that's gone on throughout the thread and all of the weird things it has done, too. I still have nightmares about this state it got me into.

8:12 This model has no respect for your computer or the resources on it. I've never had a model so aggressively just waste my RAM, screw up permissions, empty my hard drive and fill it with garbage. I had a machine with an eight terabyte drive that was almost entirely open that Astra somehow managed to fill with random disposable garbage, refused to clean up, and I had to pull an Opus just to free up enough storage to keep working on that machine. Truly obnoxious. For another example of that, here is the full disk three comparison.

8:40 He's about 15 gigs of free space on Zbook/home and one idle physical core. This machine had four terabytes of free space when I started this run on it, and it somehow managed to just destroy the machine and was using so much random resource that it couldn't do basic checks. Just obnoxious, like truly. And one last example of the model just doing and asking things that clearly were meant to be sent to a sub agent, not to the human running the whole thread.

9:08 Outside blind 69 underscore one, should the guide still send one circle for a fully shown verified original message as originally requested? It will stop at choices and keep circle disabled through blind 69 underscore one. I have no idea what any of this is asking for. I just told it to remaster Final Fantasy VII after we decompiled it. And now it's sending me questions like this. It's like the model has no ability to distinguish between what is something it understands, what is something that like it did in the thread, what is something that the sub agents should know and care about, and what is something that the human who prompted it would know. It's separation of knowledge, so to speak, here is just it's zero. It's actually zero. It has no comprehension whatsoever of what you do or don't know. They've been trying to prompt this out on the system prompt since I crashed out at them about it. As I mentioned, I wrote a long essay to OpenAI about this that I'll read and fall at the end. I actually went and analyzed the system prompt before and after cuz as I mentioned, they changed it really close to release. The biggest change to the system prompt is when Codex should stop working. The public prompt already asks it to persist. Both signed and release prompts put the rule up front and make it concrete. If an authorized step remains, take it. If a command times out, diagnose it, try a permitted recovery, and finish whatever could still be done. They also tell Codex to write a final answer that makes sense even if you skipped every progress update. They both got that fun addition to the prompt where it says it can help build original games with similar mechanics to existing games, but it should avoid recreation of a specific game or its assets. What I'm trying to say here is this model is not great to talk to, both because it will often talk to you in ways that it should probably be talking to its sub agents, and also just because it doesn't write English great. It does seem to be better than it was during my earliest testing. Like this is mostly readable. It also crashed like three times after I had it do the update here. But I am curious if I ask the same question that we had there.

11:02 This is what the same request sent to Opus looks like. For reference, signed and release prompts are a new rewrite. The public model JSON prompts almost the same as the EA prompt. Public versus signed in, release Soul and Astra. Signed in version pushes harder on autonomy. So clear. There's a new complete user's task section at the top. The autonomy section was rewritten. Permissions section merged in autonomy. Ask before destructive or irreversible actions or before sending something externally is now said directly. Final answers rule. New text. Write the final answer fresh for the reader. Do not copy reasoning, shorthand, or jargon from tool calls or sub agent briefs. This is a section that was added because of my crash out. I actually know that.

11:42 Happy it's there now, but the fact they had to do that is insane. They put in some changes to the design as well. Fonts are now suggested, not required, because it was making things pretty inconsistently roughly there. So hopefully that explains why I take so much issue with the documents read less like AI part here. Yeah. How about web design? Well, as you can guess, our friend Dara for which ai.dev is on the case and already has 6.2 Astra up. And Oh god.

12:13 Yeah, it's rough. Without the design skill, it does appear to be slightly better, but a lot of them just have really cringey image gen. It loves to do that like probably too much. I will say the new OpenAI image gen models are meaningfully better and kind of underrated. I still think image gen overall is pretty cringe personally, but background here is not as bad. Like there are things here that are not the worst, but it's still bad in comparison to like Table 5 2 with like the nice animations for these types of things. Like this is significantly better than anything we've seen OpenAI models make ever.

12:52 Yeah. So better, usable, tunable and steerable, which is nice. But the this model does have a regressing to the mean problem that we'll get to when I talk more about like the 3D and game dev side of things. Yeah. Back to the official announcement here so we can look at the scores that they wanted to share. On Agent's last exam, which tests multi-step professional work, Astra sets a new high scoring 5.2 points higher than GPT-6 Astra did at less than a third of the cost, a 5.5 points ahead of Claude Fable 5 2 at just 5% of the cost. Yeah, Fable 5 2 apparently cost 60 bucks for this and got a 72.8% whereas Astra got 78.3 at $2.89.

13:31 It also scored worse on Max, once again confirming my favorite theory that Max reasoning effort should almost never be used outside of bench maxing and even then probably shouldn't be used. We believe 62 Astra is the best model in the world for scientists. It sets a new high on terminal bench science which tests how well models can carry out scientific work in a terminal. 62 Astra scores 9 percentage points higher than 6 Astra at less than a third the cost. For what it is worth, this does seem to be true. If your goal is to use models for science, it is likely that Astra is the best science model ever. There are things in here that are sus. For example, in the Val Science Index, X High scored way, way lower than High did, 68.1 down to 64.8, which is roughly the same gap that Fable has. It's also of note that the Fable run here is on high, not X high or max, and I'm not sure how it would change for Fable 52 if you were to run on those higher values.

14:27 So, yeah, not sure how trustworthy this bench or even these numbers on this bench are. This seems like a bench that's kind of noisy when you see patterns like this in the numbers. Turtle Bench Sciences works as an overall but still had X high dip for some reason and Fable also scored quite well here but still slightly lower. It's also worth noting that it appears Astra was evaluated not using mini sui but instead using Codex whereas Fable's run was on mini sui which is a really harness. So it's possible that these numbers aren't necessarily accurate as well. Again, I'd have to dig into it more to be sure but I've seen these specific types of mistakes enough times that I don't want to blindly trust these numbers. The next section is more focused on GPT 62 Soul. As I mentioned, I've been hearing from people at OpenAI that they're not using Astra as much as 62 Soul and they've mostly been using 62 Soul in Ultra Fast. They even say here many employees at OpenAI are using Soul as their default model for both technical and non-technical tasks. I have not played with it that much so I'm going to skip this for now. I just thought it was interesting to know how many people are preferring Soul to Astra even internally at OpenAI. The next section is about law and professional outputs for documents, slides, etc. And I did some experimentation in this myself that I will show. They have these sections about creating slide decks for merch concepts and here there's an example that used 6 Astra that yeah.

15:54 It's fine. Then there is this one that was 62 Astra and it's better but in my opinion still qualifies under fine. I was curious how a certain other family of models would do here so I had Fable do its own run. One thing of note is that Fable did not have access to image gen tools so it had to create all of these itself as SVGs. And in And in opinion, despite the fact that it couldn't generate images, it actually did better than Astra. It does have a lot of cheesy clottisms like scored 100% on the test. It has seen the test before. Yeah. Did we get any of these cringe subtitles in the OpenAI versions? No, we didn't. That might by itself actually swing me back over cuz I've learned to hate these useless subtitles so goddamn much. You know what? Yeah, just for not including those, I'm going to change my mind here and go back and give the win to Astra. I even have a thing in my System Pro about never doing these and it still did that. So that pisses me off.

16:54 The next section is about computer use in interactive web design. And I will say computer use with this model absolutely cracked. It flies. It's really good at like knowing what's going to come next and batching a ton of things in one call. So it almost just feels like it flies through your machine. It's It's really nuts. I don't know why they immediately go into CAD and 3D stuff in the computer use section, but they did. But at least you could see here it does some pretty crazy stuff with CAD and 3D tools. I know Ben in particular went really, really hard with this. I had slightly more mixed experience with the 3D stuff. I'll show a bit of that momentarily. But it is impressive.

17:33 Somebody just like took a photo of some buildings they liked and had it designed and then helped them build a new light based on this building in Chicago that they liked. That's pretty cool. It is really good at cybersecurity stuff, which is a layered thing. They're working to bring GPT-4 62 models to Daybreak. They're not there yet. So whether or not you'll be able to use these models for cybersecurity is a open question. But according to my friends at Vercel, these models have already shown to top every cybersecurity bench that they have. So yeah. Take that as you will. Then there's the safety and alignment section where they talk about having made the models safer than ever with their strongest safeguards to date.

18:09 They've tightened restrictions and strengthened the system that they used to monitor models and intervene when needed. I have seen a meaningful number of people reporting getting blocked. I have not had any blocks or hit any walls with it yet. I will be sure to let you guys know if that changes. Historically, I didn't hit them too often during early access, and then sometimes I'd hit them more later. So far, post-EA, haven't hit any yet, but I'll be sure to let you guys know if that changes. And then we get to pricing and availability.

18:33 TL;DR here is that 62 Soul is the same price as 61 Soul, but Astra got a little bit cheaper. It's now $8 per mill in instead of the previous 10, 40 cents for the cash input instead of the previous dollar if I recall, cuz it was only a 90% discount. Now it's 95, and then $40 per mill out where it was 50 before. Very good changes all around, in particular cutting the cash input cost by 60% is a huge win.

18:58 So let's look at more of what I've done with it. Oh god, no. This Eldritch Horror is still haunting me. I thought about this screenshot every day since I took it. Ugh. Let's look at anything else. Obviously, we have to hop over to Fish Slop. Still absolutely full of GPT UI-isms, all the unnecessary subtitles and the like things around the screen here that just take up space that should go to the game. Like this just takes up so much space. The controls are also weird, not bad, just strange.

19:32 But like I'll be real here, it's stunning. It's one of the best-looking things I've ever seen in my browser, much less games generated entirely by an AI. These placeholder fish look like placeholders for sure, but the real ones, the ones that are part of your aquarium here, for the most part are really good. Those two are clipping into each other, it looks like. They used the wrong buttons for things like going up and down. I have my issues with this version for sure, but like graphically it's great. Once again, if you want the absolute best, it's probably a combination of using an OpenAI model for asset gen and 3D asset creation and then using Anthropic models to actually make the game pleasant to play because yeah, OpenAI still not figured that one out yet. God, yeah, I really hate this, in particular like the UI shifting around you click things. It just ugh.

20:21 There's so many of these little things, and to be fair, a lot of these like anthropic models are guilty of too. I'm just so much less tolerant of these little like broken UI things now to put so much time into something like T3 code. When a model shits out these like very unpolished edges, it just irks me. The data's there. Just train them better. But I'm going to be real, Fish Slop was not enough to push this model. We are now past the point where Fish Slop is the thing that makes sense to see what models can do. I do not have a trivial way to run the following things on this computer, but I did take videos. I started putting this model to work in real games, as in doing things to real games. One of those is one of my favorite Pokémon games, Black and White 2, but I wanted to do something very different with it. I wanted to make a ROM hack for Black and White 2 that turned them into Ultra Sun and Ultra Moon with a few story changes. For those who are familiar, these are Pokémon games. I'm a big Pokémon nerd. The DS Pokémon games were the best the games ever felt, but I have a very sentimental attachment to Gen 7, which was on the 3DS, and I wanted to bring them to an engine I considered better by backporting the new game to the old engine. And here's what I got.

21:40 This so impressive. I thought I was going to be so hyped when I saw this, but then I started playing a bit. This is after a bunch of fixes that I prompted. Like I went back and forth with the model for days on this. You can immediately see all sorts of issues with like the sprite here being clipped into the wall. Really bad sprite work here where like this one in the middle, you can't even see the pupils for his eyes. Like I know it's a DS, you're limited on pixels, but if you're going to do overworld sprites, I didn't tell it to do what it chose to do. You got to do a better job at it, man. They fixed a lot of the readability of the sprites cuz previously they didn't have outlines and it was impossible to tell what was a sprite and like where it ended. Everything just looked blobby and blurry. It was rough with the sprite pixel work. This starts to show where the limits of the model are, because while it can generate images and the images it generates can be really good, it doesn't have the capability to generate pixel art in exact specific resolutions without doing weird scaling that breaks the pixel art.

22:37 You really got to work in the art form, so to speak. Like when you have a specific pixel grid, you got to make everything for it. This model can't do that. And as I played more, I found more and more of these edges. Some of the animations it made for the Mon that weren't in this game were rough. It found a lot online too, which definitely helped it. But yeah, overall playable, but not passable. Like most real ROM hack fans would immediately smell everything wrong here and throw it out. But the fact that we are now close-ish to being able to just vibe code a full ROM hack to a game is kind of surreal. I think we're maybe a few model drops away from something like this not just being viable, but being good. So again, I had to push harder.

23:23 I'm a pretty big fan of Final Fantasy VII. I will say I have not found the remakes as enjoyable as others. I get it. They're awesome. I'm sure they are fun. I just don't like the combat system that much, and I long for a higher res and slightly tidied up version of the originals, which is why I decided to do just that. I found a mostly complete decomp of FF VII and then finished it. And once I finished it, I started re-rendering and rebuilding a bunch of things on it in order to try and make it look less awful on a modern machine.

23:52 When I first had it working on this, it started making some really really cringe AI gen images and just stuffing them all over the place with flickering from the original asset to the new asset constantly. You'll even see a bit of that here. Watch Cloud on the right there when I hit play. Did you see that? Right there where it flicks into the new asset, but also breaks the shading on said new asset. So even though it is higher res, it looks worse because it doesn't fit the environment properly. Here a particular shows just how bad that gets.

24:26 The left is the original and the right is my new high-res version. The original is 240p minus 720p. It's just worse. Like obviously so even. The left one is blurry, but it's blurry intentionally and it has a specific vibe to it. The right one is crisp for the sake of being crisp and it loses all of the subtlety that the original had. Like I'm not expecting it to be perfect, but man man this one has a vibe and this one has more abs.

24:59 It's not reasonable. It's just not that good. I I was expecting a lot more with this one. I thought this would be relatively easy, but the majority of the asset work it did it just it just didn't really impress me. I'd go as far as to say the left one looks better here because it's grittier and the grittiness is an aesthetic of FF7 that we lost in this port. But none of this is the thing that impressed me the most, at least initially. We have to talk about Outer Wilds. After this we're going to do my crash out rant and the letter I wrote to OpenAI. But before we talk about Outer Wilds, I need to talk about this vague post I made a few weeks back. I just said bellwether. A lot of people thought this was me leaking a new model name or something. Apparently there's some model internally at OpenAI called bell, b e l. I'm not aware of any of that. That is not what this is referencing at all. If that is related it is purely accidental. I said bellwether because the model made a level in my favorite game of all time Outer Wilds and named it bellwether. For those who are not familiar Outer Wilds is one of the greatest pieces of art ever made. I look for every excuse I can find to talk about it and the less you know going in to play it the better. It is an exploration game all about discovery and curiosity and I've been chasing the high from that game ever since I played it for the first time back in 2018. I I love this game dearly and for me one of the like real bars for AGI is the ability to add something to a game like Outer Wilds that feels similarly creative and curiosity inspiring. And I'll be real models can't do it yet, but holy this model got close enough that it made me a bit emotional. The game's all about exploring a solar system in space, so I figured the easiest thing to have it do would be to have it make a new planet. And it made this, a water level that was in many ways stunning. It had tides that went up and down throughout the cycles. It had music and bells and themes and ideas and no real payoff whatsoever. That's one of the biggest things Outer Wilds does right is it rewards your curiosity.

27:04 This new planet failed to reward my curiosity so badly that I had an hour and a half long back and forth with the model trying to figure out why I was stuck, just to realize I'd already gotten to the end and didn't even know I was at the end because there was no real payoff there. So while aesthetically it gets parts right, that aspect it got wrong. And god damn, the copy. The copy was so bad, like not even close to what I was aiming for or what the game originally had. I want to make this clear, if you haven't played Outer Wilds yet, none of this is going to make any sense. Generally speaking, if you watch my channel and play video games at all, you should play Outer Wilds. It is unbelievable. You might not be the type of person that it hits for, but if you are, you will be very thankful you did. If me ranting about it this much gets one person who might not have played it before to play it, it was worth the rant. This game changed me. One of the things that's exceptional about the game is the writing because it was written by the sister of the director who were very close and understood each other and like what they were going for really well. So the dialogue and the way things are written is very important in this game.

28:09 And like the the biggest thing is that the text doesn't waste your time. If it is there, it means something. The text this model wrote, despite getting the specific instruction to analyze all of the dialogue in the game and try to match it in terms of tone and character like actions, just failed outright. On the chart, I drew a boat at the bell. That boat is a guess. The bell is real and there is no my writing at its foot.

28:35 Start there, not with my guess. I want to say things that would get this video demonetized about this text. It is offensive to be as a fan of the game. This was after I told it multiple times it has to make the text feel more like the original. It just outright failed. I pulled in Muse and it did better, but it still wasn't great. Child Ellex waited upstairs while the gallery flooded around Rue. Both knocked, but Ellex's mistook Rue's one tap for the sound of the sea. Beside the hollow bell, the traveler learns why two knocks ask for one quick answer. The real disagreement is whether Ellex should come looking if Rue misses it. It's still terrible. It feels like AI robot still by a lot. Less bad than what we got out of the new Astro model, but still terrible.

29:18 Now I want to shout out a particularly cool kid. This guy's name Sakath. I noticed him in my replies talking about model personification, trying to make models talk and act more like specific people or characters and doing things to improve the writing outputs of models. He runs a business called The Intention Company that seems to be relatively small and new, so I just wrote a DM to this random kid with 90 followers saying, yo, got a weird challenge for you. I can't give you more details, but I'm adding a level to this particular game. Would you be down to take the dialogue and see if you can make it feel more like the dialogue from the original? Apparently he was a fan of the original and said, yeah, sure, I'll do it. Got back to me in a day or so after I sent him all the assets. I I took a lot longer to get him things than he took to get me things. And despite this random 19 year old kid in his small company being the ones to do this, the dialogue that this wrote was comically better. From Chert, Slate, I found your wrench. It was in the ship the whole time. The ship unfortunately was underwater. I put the new berth rather high up. This reads just like the original. Oh, and I was also strapped in to come home when I felt two knocks through the boards, so the engine's off again. Set my little scout right into that old bell. There isn't a thing in there to swing against the rim. How am I supposed to leave that alone? Oh, actually on point. This is not a sales pitch or anything. I'm getting no kickback from this kid. I was just really impressed that for once, when I DM'd somebody yo, want to help with this thing? That their thing kicked ass. So yeah, Saketh, if you're raising money, let me know. I'm going to throw a check in. I did not think you'd be able to pull this off, but you did. And to the labs, maybe acquire this kid. He managed to make the model speak English better than any of y'all have managed to do. It only took him a day. Genuinely super super impressed. He killed it. All that said, I do have one more thing I have to show. As I mentioned, all of this was from GPT-6 to Astra, building this new level, adding new puzzles to the game, and making something that felt surprisingly almost close to the original, but still not quite there. But as you can guess, when Fable 5.2 dropped, I wanted to see how much better it could do.

31:23 And here is after Fable's pass. Significantly better water, significantly better fitting models, the planet's size and the just overall experience of everything significantly better. It still sucks at playing the game, so it doesn't like move the camera or anything while it flies around. But it yeah, it did a way better job with the actual level design and making it feel like it fits in the game. So the point I'm trying to make here is it was once again a combination of models that made this come around. I'm going to actually stop watching this footage because I have not played since Fable took over and I got the new dialogue. That's my fun reward for myself tonight once I finish filming this video, if I am not too sick and don't have to go to bed immediately after. But yeah, a combination of some random kid I found on Twitter, Fable for overall game design and tidying things up, and 6.2 Astra for general design and like 3D stuff. Seems like a pretty good combo. Still figuring out what goes where exactly. Does seem like 6.2 Astra is really good at play testing, so I might have it be the play tester and then keep using Fable as the implementer. Still figuring out the balance here. I will continue to report on things as I find them. But none of this answers the tweet I showed before about Bell Weather. The reason I brought up Bell Weather is because that planet was named Bell Weather. It was a new water level that was added to Outer Wilds that had music theme with a bell in particular being core to that theme.

32:46 So why is this notable enough for me to quote? Y'all probably know Ben, my podcast co-host. He was the one who inspired me to do the add a new level thing with new models because he was trying to get new dungeons added to Majora's Mask. Ben gave a similar prompt to me, not anywhere near exact. We it was same vibes. We never even looked at what each other sent. He asked for the model to add a new level to Majora's Mask. I asked for it to add a new planet to Outer Wilds. We both ended up with water-themed, music-themed, bell-heavy levels named Bellweather. This model regresses to the mean like nothing I've ever seen in my life.

33:20 Ethan Mollick asked Astra 62 to make a puzzle spy game using any setting it wanted, and it set the game in Bellweather Station. Elliot asked for 62 Astra to create a new monster in a game. I'm assuming it's Zelda. No, it's not Zelda. It's his own new thing here. He said it to be creative and build something scary, and it made a new monster named Bellweather. I have no idea what's going on. I hate this. I really hate the idea that what I mistook initially for creativity feels to be a hard-coded name and vague set of concepts. I knew this would happen because of how quickly Ben and I hit the same thing. This is just insane to me.

33:58 And now, with all of this out of the way, I will wrap things up with the letter that I wrote to OpenAI during the early access testing. I sent this to them while we were testing the model. At the time, it felt like people were mostly positive towards it or just not saying much. I had been privately talking with a few of my early access friends who were not as into it. Then I wrote this letter, and I noticed the tide shift fast. In particular, after Fable 52 dropped, it seemed like people were pretty done with Astra. I I wrote this letter because I was just so frustrated with how the model was behaving, and also because I know OpenAI will listen. And they did. They made a bunch of changes before release in order to address as much of this as they could. While I do appreciate it, I don't think it's enough, and I do think it's worthwhile to hear this letter in full.

34:42 Even if Tebo thinks that it's my fault for not retrying things after I did, still experienced enough of this that I will stand by every word. I feel bad about this, but I know you guys will listen and you'll use the feedback to make a better model. Slight crash out incoming. Believe it or not, I wrote this all by hand with no assistance from any LLM. Sorry for the early send. I accidentally sent it before I finished writing it. Again, I copy pasted this directly from what I wrote to them.

35:05 Anyways, I do not like 62 Astra. I would not choose this over Opus 55 for any of my work as a dev. I'll start with the good things quick. It's a great reviewer. It catches things other models miss. I mentioned this before, but in all of my evals, it had a pretty much perfect score, finding a ton of things even most models before it couldn't and never lying, which is really nice. Every other model has a habit of fibbing a bit, especially Fable funny enough. I'll talk more about how I think about the difference between these models in future videos, but I just want to point out again, this model slaughtered in all my evals. So it is a great reviewer, because that's effectively what my evals are measuring.

35:41 It occasionally comes up with novel solutions that are genuinely great. It helped me cook a new reverse proxy solution for T3 code that if implemented would save us as much as $25,000 a month. Opus didn't have the balls to explore enough options. I will probably end up shipping that in the near future and if I do and it works, you will hear a lot of praise from me about this model and if it fails, accordingly the other way. I noted that it's world class at 3D and game stuff. I will say since Fable dropped, I feel slightly different.

36:05 Computer use nuts still the case. Code quality has improved meaningfully compared to Astra, but still not great compared to Opus or or Fable. It writes pretty good sub agent prompts, but yeah, it's not enough. And you guys understand this comes from a place of love. Really want you all to make something I choose in the model picker every day. With that said, I genuinely hate everything else about this model. It's hard for me to pick which problems to emphasize because I have so many. Some of the problems have existed in OpenAI models before. I'm going to ignore those. There's a new set of unbelievably bad behaviors that make it miserable to use. First, it has no concept of knowledge gaps. When this model is working on long tasks, it can pull in a lot of sub agents and manage them pretty well. It will then proceed to randomly ask you some weird internal implementation detail that you could not under any earthly circumstance understand, much less have any input on.

36:55 I sent / goal make a new level for Outer Wilds. 2 hours in, it's blocked because it wants me to compare our 24 reviewer write permissions for its clips to these random file paths I've never seen in my life. I've never had a model do this before. 56 sold it and do this. I have no idea where this behavior came from, but it's horrible and makes me never ever want to have it drive long haul work. It has made it significantly harder for me to get the model to do anything cool. It's forced me to set up other agents on crons just to check if 62 Astra asked another stupid question and blocked itself. Is this happening more because I'm using T3 code? Maybe, but unlikely.

37:33 I tried it in Codex just to check and it still had the same issues. I will say the recent system prompt changes have helped here and I also shipped a ton of changes to T3 code to try and help here too. Should be better, but still happens and still sucks cuz it just the agent will block itself over things that are nonsense. The next section is a personal favorite. It speaks in tongues that only 62 Astra understands. This model has the most unparsable outputs of any LLM I've used this year. I've shared examples before, but it just doesn't know how to explain a thing after doing a thing.

38:03 Every single explainer I get to give is so deep in the details that it borders on incorrect in the response. It misses the forest for the trees and then says the trees are the only living creatures in the forest. It's so hard to describe, but it happens almost every time I give it a complex task. I 've already built the habit of switching to Opus 55 at the end of a long 62 Astra run just to get a readable summary of what it said.

38:26 Without that, it is genuinely impossible to parse what it did on anything beyond simple bug fixes. The worst part is that it seems like you can't prompt this out. I tried system prompt adjustments. I begged it to explain like I'm 5. I asked it why work trees were slow on one machine but fast on three others. We did everything from benching P and PM to re-imaging the machine on a new file system. I eventually said, I think I'm going insane. Can you just tell me why it's so slow?

38:51 62 Astro wrote four paragraphs about why PNPM internals are working certain ways. Opus 55 said it's because the SSD is slow. I will say since the system prompt changes, I did rerun this and the answer is better. Still not great, but better. Cup two, which is the name of my VPS, takes longer to create and link lots of small files. That's why PNPM spends most of a warm install doing. The package is already downloaded and the dot repos it's already skipped. Biggest cause we could isolate is cup two's slash work storage path. Still not clear enough but where the problem is. It talks about XFS way more than it should here. It calls out some RAM stuff here as well, but none of this is just saying the answer here, which is virtual storage is way slower. The other machines had a dedicated NVMe drive. That is the answer to this question. To be very, very clear, the answer is the drive on the VPS is slow. The drives on the other VPS are direct links, which are much faster.

39:42 That's why the difference exists. It just couldn't say that for some reason. I don't know why, but it couldn't. Yeah, better but still bad. The next part is the most egregious. I didn't even remember this one because I think my mind's tried to block it out. It runs in loops and goes nowhere. I threw this at TS Rust, which believe it or not is a project I originally started in order to push the limits of what OpenAI models could do when I had early access, and they all failed to port TypeScript to Rust. So I eventually had Opus 55 do it, and it did it very quickly, and it got to the point where I could use it in real projects. No OpenAI model could even break ground on this, even after $300,000 plus in tokens were spent on it. I decided to throw Opus at it, and it was done in under two weeks for around $20,000 in tokens. This is only $500 in subs, by the way. Very, very doable. I was super excited to throw the new Astro at this to see if it could burn down some of the gas between TS Rust and other Rust rewrites. It ran for seven days and landed zero improvements.

40:34 It got super fixated on one part of my slash goal, which was make it smaller, and ended up making the compiler 56% slower in its attempts to make it smaller. Remember, the goal was slash goal, make it smaller and faster than Bun check and TSRS, which are the other two Rust-based TypeScript compilers that matter. It made it slower than it was before, meaningfully smaller, but slower, and didn't land a single compiler change. It only landed bundling changes to try and make it smaller while making it way way slower in the process.

41:05 I told it, stop working on size, instead burn down the performance gap. It ran for four more days and landed at zero changes. I then switched back to Opus and it landed a ton of changes in the next two days that made it 50% faster. Yeah. This model is not capable of patting its head and rubbing its tummy at the same time. It can't do two things at once, and if one of them might hurt the other, then it will kill the other.

41:29 It is just not capable of completing complex work that is multifaceted. The model's clearly super smart, but it's insufferable to work with on real projects. It feels like it just lives in its own world and randomly decides what parts of its brain are shown to us. On long tasks, it blocks itself for stupid reasons or runs in circles and gets nowhere. On short tasks, it writes fine code and then explains it in terms more complex than the code that it wrote. I spent more tokens parsing its outputs than I spent making the changes in the first place.

41:56 It regresses to the mean far too often. This is where I talk about the bell weather thing. I then predict how the launch will go. I said day one, people will love it, Twitter explodes, all the crazy one-shots, people will make it port projects from one language to others or finish decompiling legendary projects that have been stuck for years, be used for awesome stuff and people will be hyped. Day two, we'll see the turn start. They'll see how much usage it burns and how much less it gets done compared to Opus. They'll start merging bad code because they couldn't understand what 62 Astra said about it, but trusted it was fine. They'll see loops they forgot about running since launch that have made zero progress. Day five, people will mostly be over it.

42:29 There will be viral posts saying is Sol secretly better than Astra? It's funny, this happened day one, this happened at OpenAI even, which is hilarious because I hadn't even tried Sol yet when I said this, and we're already seeing this. Did Astra get dumber already? Et cetera. By day 14, the model will be forgotten. I even end saying that I really hope I'm wrong about all this. I'm likely overreacting. I'm admittedly known for that. I could have chosen to calm down first, wait a bit and give you a more measured take, but instead I'm dumping this here because I want you guys to know how I actually feel about this model. And there you go. You now know everything OpenAI does about how I feel about this model. This is the most transparent I've ever been in a video about my early access experience and the things I don't like because I want you guys to see as deep into my head as you can about all the frustrations and rough experiences I had with it. For every demo that I tested where it worked, I had two that it failed and pissed me off too much to even remember to include.

43:19 This model isn't just spiky. It almost feels like it defaults to bad with random spikes into good and sometimes even great. There are definitely places where this model will blow many people away and will genuinely change what we can do with LLMs. None of those are things I do. I I hate this model. I do not like using it and I don't plan to use it a whole lot going forward outside of code review and maybe as a member of my council. This is not the comeback I was hoping for at all. And honestly, I want OpenAI to do effectively a snow leopard style release where the next model doesn't have to be any better or smarter. I'd be okay if it's dumber. If they can fix these behavioral quirks and get it to stay on task better in ways that don't measure great in benchmarks, but benefit us in real world use, yeah. I would love for the next OpenAI model to not see any improvement in benchmarks, but just be better. This is a tough one to wrap. I I had a lot of thoughts during this early access window, more than I usually do and more frustrations than I ever have. I hope that all came through in this video and also that I do still really deeply respect OpenAI. The fact that they would let someone like me come in for early access testing, blast the hell out of them for not liking the model and then publish a video like this while also making sure I know that this will not affect my early access status is awesome. There are very few companies that I have that type of respect for and even fewer that would be willing to let someone like me do this. So shout out to them for that. It's probably the single thing giving me the most faith that these problems will actually be fixed.

44:44 They are lessening, they do care, and I am really hopeful that future models will more deeply consider these frustrations that I and many others have. It seems like pretty much every early access tester felt a lot of these same frustrations that I have been sharing here, and I do genuinely really hope that they can fix these going forward so that I can once again tell you guys how great OpenAI models are for your real world dev work.

45:06 But until then, I think I need to go open another Claude account. Yeah, that's where things are at. Let me know how you all feel about this one and if you want me to do a more deep dive on 6 2 Soul as well as everything else you think about this model. There are definitely going to be more comparisons coming up. I'm excited to do a tier list finally now that I think all the big releases for the year are done. I'll let you guys know when those are all ready and more. Until next time though, peace, nerds. I also just realized I'm wearing a dress shirt and sweatpants.

45:36 I don't think there's any more apt analogy for this model. I'll leave it there. Leave this in, face.

Summary

OpenAI's release of models 6.2 Astra and 6.2 Sol has generated significant buzz, especially as they aim to compete with Anthropic's recent successes in the AI code space. While Astra shows impressive capabilities in complex tasks and is cheaper to operate than its competitors, the reviewer expresses frustration with its usability and communication style, suggesting that it often fails to provide clear, actionable outputs.

- OpenAI's 6.2 Astra and 6.2 Sol models are positioned as strong competitors in the AI code space, especially against Anthropic's offerings.
- Astra demonstrates superior performance in complex tasks at a significantly lower cost compared to models like Fable.
- Despite its capabilities, the reviewer finds Astra difficult to work with due to unclear outputs and a lack of understanding of user knowledge gaps.
- The model's communication style is criticized for being verbose and convoluted, making it hard to extract useful information.
- Astra excels in 3D design and game development tasks, but struggles with generating pixel art and maintaining context in longer tasks.
- The reviewer highlights the model's tendency to block itself with irrelevant questions, hindering productivity.
- OpenAI has made adjustments to the model based on early feedback, but the reviewer remains skeptical about its overall usability.
- The reviewer expresses a desire for future models to focus on improving user experience rather than just performance metrics.

Questions Answered

What are the key features of OpenAI's 6.2 Astra and 6.2 Sol models?

OpenAI has released 6.2 Astra and 6.2 Sol, which are seen as significant competitors in the AI code space, especially against Anthropic's recent models. Astra shows superior performance at a much lower cost compared to previous models like Fable.

What problems does the model face in understanding user prompts?

The model struggles to distinguish between different types of knowledge and often generates irrelevant or confusing questions, indicating a lack of comprehension.

How have the pricing and availability of the new models changed?

6.2 Sol maintains the same pricing as its predecessor, while 6.2 Astra has seen a price reduction, making it more accessible for users.

How does the model perform in generating text that matches the tone of existing games?

The model fails to replicate the writing style and tone of the game 'Outer Wilds,' leading to disappointing results that do not align with the original content.

What issues arise when the model is tasked with long-term projects?

The model exhibits a lack of understanding of knowledge gaps, often asking irrelevant questions that block progress on long tasks, making it difficult to achieve desired outcomes.

© transcribe · For agents Built with care and craft by Gokul Rajaram