Transcript
0:05 Hello everyone. Welcome back to Semi analysis weekly. This week I'm joined by Matt and Andrew, the authors of the very controversial but very interesting anthropic subscriptions offer five times or more value than OpenAI. This was an article that we put out a couple of days ago where the guys did limit testing for every AI subscription plan. It's not just open AI with codeex and anthropic with claude code. It also includes Meta, SpaceX AI, Miniax, Moonshot, ZAI, Cursor, Cognition, anybody where you could be getting an allin plan from. and we're going to talk about the learnings that they've had. Guys, welcome to the show.
0:45 >> Thank you for having us, Jordan. >> Okay, so first TV on something like this is too polished now. I'm not used to it. Well, yeah. Okay. Other people need me to like direct them with questions, but for you, I think we can let you run it. Max, the plans are highly subsidized. No. Well, okay. It depends on what you actually depends on how you want to sort of define subsidation relative to API pricing. The plans are obviously a better deal. I mean, we can throw up some graphics on the screen here or maybe people can read the newsletter if they haven't, but some headline results including you being able to pay like $200 for a Cloud Clan and get like $12,000 of this 55 tokens at API pricing. So, that's obviously a good deal. however, in terms of whether or not the subscriptions are like subsidized in an absolute sense, like do you get does Enthropic have negative gross margins when they're selling a subscription plan? I think the answer is no. the reason for this is that a they're margins at API pricing are incredible. I think consensus it's like over 90% blended at this point. And the second thing is that most people don't use like 100% utilization. And so if you combine like a low sort of realistic average utilization or I think it's around like say 10%. with this you know 90% plus margin API pricing you end up with subscriptions having like call it 50 to 60% like blended gross margins for anthropic. so that's obviously still actually a pretty good business for them, but compared to their like 90% plus margin API business, then subscriptions are still.
2:34 >> So maybe the takeaway is not that the subscription plans are subsidized, but that the API pricing for the current Frontier models is like, you know, punitive, exploitative, price gouging, whatever word you want to. >> It's quite expensive. It's they're selling to willing buyers at the fair market price. This is capitalism. >> Yeah. >> Understood. Okay. >> So, talk to me about the 100% utilization and how you actually did the testing to realize these like numbers, these results.
3:10 >> That's a that's an Andrew question. >> Yeah. Yeah. >> yeah. So I guess to figure out like the results for the subscription plans were pretty much just like we bought every single plan and ended up measuring every single token type. and from that we're able to say what kind of workload we are running. In this case generally agentic that's pretty much what I've been using these subscription plans for. And they all tend to be around like 96ish% cash reads a little bit of inputs and a little bit of cash try around 2 point something percent and a little bit of outputs. you blend that together and then you can get the big numbers that we figured out. Oh yeah, sorry.
3:53 Before you dive more into the details, I do want to emphasize that there have been people on Twitter before who did similar analysis and they would essentially like just present one dollar value as like the equivalent, you know, API value of subscription. And that is sort of just a fundamentally oversimplified incorrect analysis because the way to think about your subscription plan is that by giving the company, you know, $200 a month or whatever, they allocate you some number of like credits. This is just some other like, you know, currency. And you should think of each model token type combo as sort of costing a different number of credits and essentially drawing from your like allotted user a different amount. And because sort of the ratio between different model token type like credit costs can be different from their API price ratio. the actual like API equivalent value of your subscription plan can change dramatically depending on what model you're using and what workload you're running. And what this means is that like you can't just make a blanket statement like oh the $200 enthropic plan gives you $10,000 of value. You have to say like the $200 anthropic plan when running a particular workload on a particular model gives you like some dollar amount. And this is sort of the the nuance that we capture in our newsletter and something I really want to make sure that people understand.
5:23 >> Yeah, makes sense. Can you go even back to basics even more and just describe the different token types as you're referring to token types? I'm not sure everybody might understand what that means. >> Yeah. So when you're when you're in your cloud code or your codeex app, there there's different token types that are being used there as you chat with the model. the first one is the input tokens. input tokens are uncashed. they're just fresh tokens that you send to the model. when you're actually doing like an agentic multi-turn conversation, most of your input tokens are not from your direct chat but are from things like web search. where the fetch something and then analyze it in like a small sub agent in one shot. in that case, it doesn't need caching and so that's where it won't use caching and it'll be like a fresh input token cost. the second one is a cash right. cash right is in a multi-turn conversation rather than reprocessing your entire prompt every single time when the when you call like the LLM API server you can save the previous results into a cache and reuse them later for faster inference and also for cheaper costs. And so cach write tokens are previous tokens that you write to the cache and they'll they'll build once when they're written and then once you read them back they're built back as cache reads. and the nice thing about cach reads is there is that they're extremely cheap because they're you're essentially just serving from like the KV cache on the LLM server.
7:04 so yeah so now you have inputs cache writes and cache reads. input is like generally let's say like 1x a 1x price. cash rights are generally like 1.25x of like inputs. cash reads are like around.1x or less of of input costs. And then lastly you have output tokens. So this is outputs is what the model actually generates. this is generally the most expensive token type on Fable and Asterits around like $50 per million token compared to like $10 inputs. yeah, those are >> it's pretty straightforward to draw a line from like the token type to the actual processing that's happening on the accelerator that's producing the tokens. In other words, reading KV cache from memory would be a cache read.
7:56 This is a lot cheaper in performance terms than outputting new tokens which would be decode steps over the model, right? >> Yeah. >> Yeah. Correct. And cach read is like probably the most important thing and like the way you do caching with LM because you know LLM are they're like auto reggressive like coders. if you don't do caching, you're like reprocessing the entire prompt each time which is extremely timeconuming and expensive. and because most of your chat session is actually catch reads because again you're reprocessing the entire thing every single time. So if it's cached all of that is essentially cash reads. you want this cash read price to be as low as possible because it almost dominates the price of your chat.
8:40 >> So can you talk to me about the like the technical way? >> I just want to say this this is like teasing our our endpoint X testing a little bit. But really having like a high cash ship percentage is what separates like the good infant survivors from the dosh ones. And this is like actually surprisingly you know difficult problem to solve but it makes a huge difference in terms of the actual cost to like run the same workload for sort of everything was saying about the different token prices.
9:09 >> Yeah. And even open router recently added like if you go to open router on any any given model. If you scroll down past the top level you know prices per provider they have a nice effective pricing which showcases like what the effective like cost a blended token costs are assuming like certain cash hit rates of like the providers. It's pretty neat. >> Yeah. Okay. So explain why cash hit rate or CHR is like a really important metric for people who work on inference serving today and why it's actually hard like we've been saying cache cache is like conversation history right so why is it hard to route a new request to where that conversation history is stored or to tar it off to lower performance but cheaper storage.
10:05 >> Yeah. Yeah, if you're running a real sort of production inference workload where you have like multiple different racks, multiple different deployments of like the same model on each of these racks, and then each one is processing like a batch that has tons of different user requests. Essentially, like when I submit like say the first turn of my user request, you have to be smart enough to like write all those tokens into the cache somewhere. you know probably starts off in like staying in HBM if you're pretty confident I'm not going to sort of spit a follow up that quickly but then if you know there's a large gap you probably want to sort of offload it to a lower cheaper memory key there see there and then when I come back with sort of like the next turn of my request obviously you as the provider have processed I don't know if you're big maybe thousands tens of thousands like millions of other requests in that time delay but you have to like take my new request probably send it to like the same deployment of like you know the model that you have sort of find my previous you know tokens that you hopefully like wrote down to cache somewhere and then sort of route those to the same deployment and then sort of like reprocess the next turn and I think this this process generally of like sort of having to find the previous cache like routing like the the subsequent turns of the same request to the same deployment sort of balancing everything properly so you know none your GPUs are like being overutilized and you don't have a ton of GPUs are just like sitting there idle burning money. These are all very complicated systems engineering.
11:40 >> Yeah. Yeah. And there's also this TTL dynamic too, right? Where like every new request is going to be tagged with a time to live for. So it's going to sit in HBM the highest tier maybe for 5 minutes and then the next one maybe for an hour or something. But it's a punishment to the user if you let your old conversation die. so that it's no longer cashed and now you want to keep it going, you got to pay a whole fresh input cost or cash right at 10x the price or something to be able to rehydrate it. So often times more than 10x at this point >> kind of crazy.
12:17 >> Yeah. Actually, one of one of the things like one thing that we called out, I think originally to our Tok subscribers a few months ago at this point is when you looked at like you the original enthropic API price ratios, it was like if cash reads were one then input tokens were like 10 and then output tokens were like 50. It was just like 1 10 and then five more ratio. And one of the things we've like noticed recently with the price cuts that open eye and dark have done is they've been overwhelmingly concentrated on making the cash right the cash reads cheaper. and why is this the case? Well like one as we've established this is like the most important lever to pull to actually decrease overall effective pricing. But then two, and this is a thing that we pointed out to our tokconomics subscribers a few months ago, is that cash reef started off at as like most likely by far just the highest margin token type amongst those three. like it's very possible that following this original price ratio, Enthropic was ripping like 99% margins on their cashers, which is just what and then now we're finally sort of seeing those prices keep competed down again, you know, thanks to just competitive pressures. That's one thing I want to point out.
13:30 >> Unfortunately, I don't think this has reached OpenAI yet with Astra still $1 cash reads as compared to Fable at 25 cents. >> Yeah. Yeah. Yeah. Which is actually a big reason why you'll see on the plans Astra might have like higher dollar value on the Pro 200 plan versus Fable 5.1. but actually Fable 5.1 has more tokens and more usage. because its cash, right, its cash reads are 25 cents per million token versus $1, on Astra. And that's besides the whole point that, you know, Fable 5.1 is only 50% of your play and you have another 50% to use.
14:08 >> Okay. Yeah. So, so it's take me through the exact like OpenAI versus anthropic on the frontier model comparison in these plans. Like this is the headline chart I think that a lot of people who have seen this article have seen which is like how can we compare Astra to Fable 5.1 straight up on their different plans that they offer in terms of how much value you're getting on a per token basis for using these models which I mean the choice of model that you prefer it seems like a very personal choice at this point. It's not clear to me that anybody would say like it's it's certainly not for my usage. It's obvious to me that Astra is better than Fable or Fable is better than Astra. They seem to just kind of trade off with different tasks that I'm trying to do. but I Yeah, tell me about like the actual usage.
15:05 >> Yeah, so funny enough, I think this was not the chart that was getting traction on Twitter. Let me cover that one next. This one I think people weren't complaining about because it shows open eye and thropic is like roughly comparable for you know the same you know $200 or $100 a month or whatever. like one thing to emphasize though which people missed and what Andrew talked about earlier is that you know originally people were claiming that our other chart was like unfair GBT 6.1 soul because that API pricing is cheaper than Opus. Well, for this like real headline chart where you're comparing Astra versus Fable, Fable is like effectively 40% cheaper API prices than Astra or something because their cash reads are 75% cheaper and that's like 95% plus of your total tokens. And so if you were to look at it from a token percentage or or just a token point of view rather, you get like 1.9 billion tokens of Astro for the $200 a month plan, but you get like 3.1 billion tokens of Fable on the same $200 a month plan. And not only that, that 3.1 billion only counts for 50% of your limit, not the full 100%. Because Fable was restricted to only using 50% of your anthropic limit. And so you can actually use like another million billions of tokens of opus on top of that for the same $200. so I think even this chart makes it clear that Anthropic offers a better value if we were to switch to the soul versus opus chart. I think it's an even more brutal mock by Bravik.
16:38 maybe And this time. Yeah. So this is a really controversial chart on Twitter where Rex where everyone was kind of saying, "Oh, this is so unfair because you know 61 soul is so much more token efficient than Opus 5.5." And everyone really liked to bring up like the artificial analysis opus 5.5 max cost per per task and like the 6.1 soul max cost per task and it was like something like $5 or something and then 6.1 soul is like 72 cents. and so people would kind of use that to justify that oh this isn't a fair comparison. there's like a few reasons why this is problematic and like even if you were to assume that like you know soul is really that much more token efficient. The it still if you do the math like it still ends up being that like opus 5.5 is like two to three times like the value in terms of tokens. but yeah the exact math here is so one we we don't agree that like the artificial analysis intelligence index yeah is necessarily a good way to measure token efficiency.
17:47 We'll get to that point later. Even if you assume that like sort of using these contrived, often broken, often saturated like benchmark tasks are a good way to measure token efficiency. Opus 5.5 on max is like the number one model on the index and 6.1 soul is not. And if you just use like opus 55 on medium, that outperforms like all the 61 soul results except for like max. So I think specifically 61 soul on x high and like obus 55 on medium score the same on the aa hundus index and then if you look at sort of the cost per task for those two specific runs it's like a two to 3x diff or something and then you're getting 5x more you know API equivalent value according to this chart and so really you're still getting like 2x more value on opus than soul still and this is even if you sort of assume that this like broke token benchmark index is a good proxy for token.
18:50 >> Yeah. The unfortunate part though is that like obus 5.5 is not really in the 6.1 soul class. It's actually competing directly with Astra. and like >> I don't know if I fully agree with this. >> I think many this is anthropic fanboy Andrew's point of view. All right. >> This might be my point. This might be my point but I would say like five even on the you know the broken birth marks is still like number one a little bit intelligence in this. a lot of people have been substituting like Astra result as 5.5 tropics have been saying you know it's as good if not better than Fable and so like in that scenario like you have Fable 5.1 for like let's say that's the Astra competitor you have that for half your plan and that's essentially equivalent to the $200 plan already on the FGBT but then you also have 5.5 which is a beast for the rest of your plan at like $6,000 or $5,000 worth of value. And and when you do that comparison, it's like not even a question in terms of who's providing more value versus the tragic BT thet versus entropic subscriptions because open 5.5 is just like a step above section console in most categories.
19:57 Yeah, for the audience, Andrew has a repo here as semi analysis where he doesn't let a he has an agent. MD that like you need to say which model was used to break this PR and if it's not Opus 55 or Fable 5.1 he auto redempt the PR right he is not even allowed to config to his repo. >> Yeah. Okay. So, this is kind of what I said earlier, which is that this is a very, >> this seems like a very personal decision at this point of which model you're going with. And it's actually endorsing this on everyone, right? He's he's making all the internet X Astra fanboys switch to Opus after they want to intervene.
20:38 >> Hey, and they saw the light. Everyone who everyone who started doing it, they convert. That's true. Yeah, >> that works. >> They saw the light. Okay. okay. can you talk to me a little bit more about the the methodology? So like how did you go about actually measuring like per token type and how the limits fill up because I I think this is also a way in which people get confused which is that like they launch one really big prompt and then their you know hourly rate limit or five hour limit or whatever it is just is gone.
21:25 And you know, you made the point at the beginning, Max, that like nobody's using these things 100% of the time, but what's the scenario that you guys actually pursued to use this thing to figure out how many tokens you could consume in these windows? >> Yeah. So the rather than you know launching some task or benchmark and like seeing how many tokens it uses you know the split for that we really wanted to like do like a kind of a scientific experiment to like extract like the exact value for the exact rate for input cash right cash read and output token prices. and so the way we do that is essentially kind of mocking out the way that all the popular CLIs call like the model APIs essentially to mimic exactly how you would hit the inference server as a normal subscription user. And then we use like essentially two types of prompts that let you maximize the token type that you care about and like minimize the token types that you don't care about. so the for the inputs cash rice and cash reads we use like a small tag at the initial start of the prompt and then a whole block of war and peace that just like takes up a ton of tokens and then ask like a very very simple question to the LLM to minimize the number of output tokens that it generates from the request usually like four tokens or just saying like history or fiction or something.
22:58 and like the way that the the reason this works is like let's say for inco tokens we can set a cache header to mark something for caching or not and we can also randomize the tag in front of the initial in front of the prompt which essentially guarantees that nothing will get cached and also that the API is not really trying to cach it because there's no cache header on there. and this is present in like entropic and open eyes APIs and other models. and then we read back like the requests response to verify which token types were actually consumed so that we know that yes like this was all like for example for cache rights it's like a similar story to input except we do mark it for caching and so we see that the whole prompt gets cached besides the last maybe like model generated answer was like net tokens and then for cache read it's a similar experiment as well and for that one we just set the cache header one call is going to be a cache right and we just drop that from our counting and then just keep sending the same request and that gives us like cache reads on like a per request basis. and then finally output tokens is just like write the super super super long like technical essay and then just have the models like rip out 16,000 output tokens. and that kind of consumes that budget and we can like look back at the response and the meters and be like, okay, that was like 2% of our weekly limit right there or 5% of our weekly limits. Apple tokens are expensive. I probably use like 5 to 10% per one of these calls. but yeah, by just like repeating these and just kind of graphing it out like you can figure you can figure out like you do the math and isolate exactly how much each of the token types costs. and then we can use that to do any sort of analysis around like okay if you're running like a gentic workload like under this token split like this would be your like subscription value and then under like some chat some chat workflow then this would be your subscription value etc etc. talk about how you get those ratios as well. Like we're keeping good track of what a chat workload versus an agentic coding workload actually looks like, right? In terms of ratios of input to output to cache read to cash, write.
25:22 Mhm. Yeah. The aentic one actually comes from our tokconomics dashboard which has semi analysis's own internal usage for like the month of all the months and then we just use like the month of September the one of the most recent and that's that was our own token split because all of what we do on the like enterprise accounts is just agentic work for chat. It's a little more difficult because like the only way to really get a good token split for that is like Hatch BT and have like the realistic workload. but you can kind of estimate it based on the fact that you know there's a system prompt on every single chat message or like chat thread and that system prompt is essentially cached for all users of claude or chatbt or whatever and so that alone is going to be like a big percentage of cache reads and so that's how we kind of that's where we start and then you assume users are sending like shortish prompts maybe they step out and they don't respond to the thread ever or they come back and step in once or twice. And so like under those assumptions, you kind of get like this flip that we have in the article which I believe is like is it like 5% uhish inputs some 20ish% cash cash rates >> here?
26:43 >> yeah, 13% cash rates, 75% cash inputs, and then 10% outputs. >> Makes sense. Makes sense. okay. How about some of the other coding plans? one of the pieces in the subtitle or or header is that there's other all-in plans that aren't from Anthropic or OpenAI. first of all, does this actually matter to anybody? But yeah, let's assume it does.
27:16 what do you guys think of like Muse from Meta, Miniax, the Super Gro Heavy, Cursor Composer, GLM, and Kimmy and how they're actually measuring value compared to some of the other guys. >> Yeah. So, I was actually kind of surprised that a lot of the Chinese guys, so Moonshot, Zai, and Miniax offered pretty good like API equivalent value per dollar. reason why this somewhat surprised me is that they're they're all like so compute poor and I think if you look at their API pricing it's even lower margin than like the furniture labs because it's a a more competitive business like more like commodity but some of them I think I think Moonshot was the worst at like roughly 7x API value or something. but some of them were you know in the same kind of 12 to 15 range of OpenAI. and considering that OpenAI is sort of the exact opposite of compute pore and you know cheap API pricing at least for Astra. that was pretty surprising takeaway for me. But it's it's not maybe the takeaway for me on this was that it's not obvious that when you look at one of these charts that any of these plans are just like so much more compelling on a dollar value basis that you would be like willing to be from OpenAI or Claude because like >> okay I I get you know so much better value than on my Muse subscription that I'm willing to take a slightly worse model because I just like get so many more tokens. Like maybe it's two times or three times the Open AAI or anthropic, you know, upper limit. or maybe it's it's nice that it starts at a $50 fee instead of a $200 or $500 fee to like really get access to their top model, but nobody's doing a price war on the monthly subscriptions at this point from what we can tell. Right.
29:21 >> Yeah, that's true. I think a big reason for that is nobody can touch Enthropics margins at API prices. And so if they offer a similar API equivalent value per dollar, it probably means that their actual margins are like way worse than probably. and so this it really just underscores that in in the AI age like being the person who actually owns, you know, the model everyone wants to use is a huge advantage. being a rapper is especially bad. We covered this at the end of the article, but if you look at using like entropic models on cursor recognition, the limits are just like way way lower than at first party. And obviously this is like you know when you if you just think about it for two seconds like this obviously has to be true because anthropic you know has 90% plus gross margins and then cursor and/ computation probably at best have like a 20% enterprise discount or something compared to prices but it's like it was pretty cool to see this kind of claim validated by the data.
30:28 >> Yeah. Yeah. Makes sense. How about small small shout out to Meta for being one of the highest value per dollar plans on like a $50 >> subscription with Meta. You can get like $2,500 worth of like new Spark and that's like I believe it's like 140 or 260 million tokens per dollar or 13 billion tokens a month. It's quite a lot. like if if they keep the same rate and use Spark is like a little bit better, it might be one of the very >> Yeah, man. For just for that $50 plan, that's like 250 maybe even like 300 tennis reservations in San Francisco that you can have Muse Spark getting for you.
31:19 Tennis reservations per dollar. I'm not sure, man. I think >> Yeah. Yeah. okay, let me let me quiz you about this one. So, we we've put out a chart similar to this before, Max, which was I thought was super interesting the first time, and it's like even more interesting this time. We we have no visibility into what the utilization metric actually is on these plans. Like there's obviously some loudmouth power users on X that are using 100% of all seven plans that they've got maxed out. Then there's other people that are a little more reasonable maybe like myself who goes and records podcasts for half the day and doesn't use this coding agent or it works in the background installs and then I got to go you know product to go do more. I take naps. I sleep, you know.
32:11 >> Okay. I don't take naps, but >> whatever. >> You have a wife. >> Fair enough. Yeah. Yeah. My daughter takes naps. Yeah. Yeah. so, okay. What what do you think like if you had to speculate what do you think the gross margin percentage that these guys are on the coding plan actually is for the given models like like where would you put these guys is it the average user is using 10% 100% somewhere in the middle I think for enthropic it's like it can't be much higher than 10 reason for this is that if you if you really squint at all their leaked financials it just does not tie out if subscriptions are like extremely negative margin business. I think it basically needs to be like a decent maybe call it somewhere between like 40 to like 7% margin business in order for just a leap financials to all tie out.
33:14 we walked through all the details in our tokconomics model. but that that's why I think the average utilization can't be much higher than like 10%. And I think it makes sense if you consider the fact that a lot of the subscribers are actually enterprise and slash on like business plans. And I think if you're a if you're a business that's like buying $200 a month plans for all your employees and then those people are only spending like $800 a month on credits or something $1,000 a month on credits, right? You're still you're still kind of happy with that deal and that corresponds to like sub 10% utilization.
33:50 >> Fair enough. in your discussions with people who have these plans, what's their initial reaction to you saying this? Like I think initially anybody who's tried to pay per token who has a way to consume the the all-in plans is kind of like I' I'd never pay for tokens willingly because the all-in plans are are a good deal. but what's the what's the reaction roughly speaking when you've talked to people about the article? Like they kind of agree, they disagree like Do you think people are are around that 10% 20% utilization number or something like that?
34:27 >> Well, the guys on Twitter were a lot about this definitely aren't. These are like the 100% guys, you know, with five accounts >> recreating Minecraft every three days, something like that. >> Yeah. but I think their their like vibe of the plans kind of matches our raw data or a lot of people have been saying for the past couple weeks that like Opus 55 on a claw plan in particular is like the most bang for buck thing you can get. and also it's actually important to note that the 50% $200 a month the 50% cut that I made, the $200 a month plans limits that they announced at dev day hasn't actually come into effect for anyone who purchased a plan before that announcement because they got sort of like a one-mon grace period for OpenAI.
35:18 And so we were actually quite puzzled why Twitter was not like roasting openi more for just dropping this 50% limit reduction and like you know giving you a bunch of code about how like oh because GPT7 Luna is actually going to be like smarter than GB6 Astra this is a win for you guys like that was kind of dumb. but no one like calling out about it. I think that was maybe largely just because like a they timed it well with a bunch of positive dead APR and then b they gave you this one month grace period and then perhaps like a month from now the limits actually like go into effect for the majority of users. we'll see some more noise about it all.
35:59 Yeah, that was good timing by them to announce the cut at like 900 p.m. the day before dev day and then just drown it out with like 20 awesome announcements the next day with all this other stuff. >> Dude, just imagine if Enthropic announced this. Like they they would have been roasted a lot. >> They wouldn't have made it. >> Yeah, makes sense. okay. What What haven't we covered here that we should be talking about?
36:31 That's basically everything in terms of the subscription limits. I guess maybe one thing to emphasize is that this newsletter is very much a point in time and as like new models come out, new subscription tiers come out, new things like ultra fast mode come out or just you know the providers decide like hey actually I want 80% margins on my subscriptions too not just 50. We expect all these numbers to change pretty dramatically and we will be sort of tracking that continuously and then you know reporting the updates as they have >> and keeping in touch for model quality and token efficiency right which is going to going to come into play in future I'm sure >> dude it's so hard to actually measure token efficiency well like I wish there's a way to do it >> you can't rely on these benchmark tasks dude but then without benchmark tasks it's like how do you actually verify the two models were, you know, similar quality.
37:25 I don't know, man. >> Give two people analysis the exact same assignment, check back a week later, >> measure the quality of that assignment, and see how many tokens they used. >> I think I think there's a a huge you know, confounding variable there, which is the person doing the task. >> Yeah. What do what do you think is has a bigger impact on token efficiency? same person using two different models, Astra versus Fable or something, or two different people using the same model to try to accomplish strong opinions on this one.
38:02 >> I actually came in, he looked at the semi analysis token stone leaderboard, and he has some strong opinions on this one. So if you look at the leader board, >> people who use the top people on Astra and Fable are spending about similar amounts, but the Fable people are using more tokens. doesn't really tell us much about like how much are getting done though, which is the difficult part of answering this question. I'd say from like personal experience trying out like different models, it's you can definitely see token efficiency on OpenAI side. but it's like what I like to think about as short-term token efficiency in the sense that yes, I'll get like this immediate task done, but I feel like there's so many follow-ups that I would have liked to just include the Opus or like Babel would have just done and like 61 Soul or Astra might have just not done it for the sake of completing the task faster.
39:04 and so like that sense like in the long term I feel like the entropic models build you like the better like code base and like definitely more readable one. If you read some of these like Astro tests it's it's something else but and so that's like my opinion. It's just like I think models matter quite a bit to do efficiency but over like which user is actually using them. What about Andrew? What about the resellers of these things like cursor's old business, cognition, perplexity?
39:43 where do you where do you think about their influence on which model you might use? Like is the concept of a router just still never showing up for you where you have to switch models or you you allow perplexity computer or something to use a given model for a given task because it thinks that's the model to go with and it has different taste than you do. >> Yeah.
40:15 I think people I mean I don't use cursor or Devon myself, but if I were I probably would just select like my favorite model anyway. I know like you know Devon has like sweet 2 and Fusion which is supposed to use like a cheaper model alongside like a larger better model to keep prices low and like this looks great on benchmarks. I haven't personally tested it myself in the real world, but my general vibe is that if I can and like my company like is willing to pay, I would definitely just use the like an Opus cheaper model rather than like a sweet two model for example or even like a Sonnet 55 cheaper model because I just like for me like I just trust those models to like do the right thing all the time. where I might not do that for like another like unknown model to me. I think that like as you work with these models that you kind of develop like what each model's good at and like where they might bite you and like that's actually relatively important when you're designing systems.
41:24 >> Yeah. How about the rise of open models? I saw some claims the by David Friedberg on the All-In podcast that there was a flip from open to closed from closed to open models from 8020 to 2080 in the last 12 weeks. And you know, we wrote a whole article about for our tokconomics subscribers about how this is like not true because they're just looking at router data. But what I learned through there is that there is a pretty meaningful move at least across the routers and at least across token share but maybe not implied revenue quite as much that people are using open models more. So, do you have firsthand experience with this or talking to other people that would lead you to believe that something about the Frontier Labs either how they're messing with their prices or the quality of the models as they release new ones is just like not far enough ahead from the open model ecosystem that's lo leading people to try out open models and then get stuck using them over time.
42:26 >> I think it's a question of how far ahead the closed source models are. I actually think it's just a question of how good of a model do I need for my task and it is true that like a lot of software engineering white collar work in general does not need like fable 5.1 ash for 6 level intelligence and so I think a lot of businesses especially the ones who are like lower gross bargain businesses are very rationally making the decision to start offloading some of the easier tasks to these increasingly capable open source models. I think this is going to continue like happening in the future.
43:08 And so then the question of like are the Frontier Labs a durable business model like really just comes down to your belief on whether or not sort of the space or TAM of new things you can do with increasingly smarter frontier intelligence will outgrow the portion of tasks that are being offloaded to these cheaper models. And I actually think this is kind of the the biggest sort of difference in priors between people who are bullish like Frontier Labs versus people who are super bullish open source. It's like if you were to sort of magically spawn 100 million super intelligent PhDs who are experts in every field who also don't need to eat or sleep or anything. do you think the economy would be able to absorb that like relatively quickly and still get high ROI or do you think fundamentally like the world's just not that intelligent about all that? I think like reasonable people have very different prior on this. but if you you know believe that the answer is yes to that question, you're bullish, the answer is no, you probably think we're going to have like a recession, you know, relatively soon and you're really bullish like together.
44:22 >> Yeah. Interesting, man. I'm I'm motivated to ask the question because I've been using so much perplexity computer in Slack recently in particular using GLM 5.3 because of all the refusals on anything cyber security related from both open AI and anthropic were making it impossible to do my work and so I needed to use shield 5.3 and I think that led to me discovering that yeah it's pretty good for a lot of stuff beyond >> it's like a onelevel model honest right I on the cyber benchmarks. It was it was crushing it and I I literally can't test it. It to me it's a frontier cyber security model because I literally can't use the others for myself.
45:04 >> So, you know, it's it's yeah, whatever. I can only look at benchmark model cards. I can't actually use the other stuff. Even though I'm in the CVP program to get verified to to do all the cyber work, I still can't get past the guardrails. I need some more creative prompting to get past the guardrails trained into the model, not the classifier outside of the model. But anyway, waste of time to spend time trying to convince the model that my grandma's at risk and you know, I'm going to bribe it with a peanut butter jelly sandwich if it doesn't get this thing done. Just may as well switch to GLM and it'll it'll do the work. Well, what happened is like I'm in that same chat history. It's got all the data inside of it and I didn't bother switching to the other stuff to just make a PR on a dashboard repo to put out some data that we got from that that work and like whatever for probably 10x less the token consumption in terms of pricing it put up the dashboard PR hable 5.1 for you know 10 lines of code >> this is music to Dylan Patel's ears >> a lot of things do not need Fable 5.1 for sure.
46:13 >> Yeah. But I also feel like I'm using this stuff day and night. My wife would tell you I am up late using this stuff day and night and I'm not cracking the top 10 on the leaderboard. So I still have no idea what the hell you guys are actually doing with all these tokens. >> We're we're like consistently not the top 10 anymore. So really you got you got to interrogate like you know Jeremy and Kyle and Andrew Wagner guys.
46:37 >> Jeremy be back on soon after I'm finished asking him questions about data centers. and I'll be interrogating him about what this fast mode is actually doing for him and how many sub agents he he told me he just takes one time he did admit that he he he takes like arbitrary odd numbers and says that's how many sub agents to launch for a given task which does not need sub agents just be like give me 37 sub agents to investigate all the permits for this one data site >> it's a point of pride at the top of that and then he's he's like oh that was $2,000 Sorry.
47:12 >> It's research, guys. It's research. Yeah. Don't don't watch it, man. Don't don't don't question my methods. Okay. Okay. >> You know, I do want to add one extra thing about GLM 5.3, which is like after talking to some very prominent hackers recently in or actually in the world. it seems like even 5.6 six soul if you have the cyber unlocked it's a much much much better model than go 5.3 on real world cyber tasks and so while like some of these models might achieve the scores on the benchmark I think like 5.6 Cyber Astraic Cyber Mythos 5.1 actually unlocked might be a lot a lot more capable than a lot of people think compared to these silver.
48:01 >> Yeah, there is a >> which is a little scary. It is >> it is a real possibility that the people who are inside these frontier labs and and get to use these models before anybody else and understand their capabilities better than anybody else might not be completely lying and pursuing a convoluted scheme for complete regulatory capture and are actually concerned about the cyber capabilities of the model. Do you guys think that's that's a possible scenario that's that's playing out right now?
48:30 I think it's very possible when the question is whether or not they're going to recapture. No, >> I'm just giving you a >> I think you mean like the researchers are genuine like when they're like >> I'm giving you a satic thing to say. Maybe the researchers >> Sorry. Sorry. I was open just released a new repo called math where I think they just like probably made more progress in the field in the past like 10 years combined or something. So I got distracted with that.
48:52 >> I didn't hear your question. >> I think they might have just solved another millennium prize problem. I think that's a better response to my question than what I was asking, which is like, yeah, maybe the people who see these breakthroughs, you know, months before us and are really concerned about what's going to happen next are being honest and genuine about their concerns and not pursuing some convoluted regulatory capture scheme. That's all I was saying in in more words and >> that that's that's a ridiculous take, Jordan. It's obviously regular cas.
49:26 >> Oh my god. H anyway guys, I think we've derailed it enough and we need to go read an OpenAI blog post about some crazy math things. So >> d the repo on GitHub is OpenAI/Math. That's like that's high war and just a bunch of mean >> we're get serious. >> And then and then like 10 columns I've never heard of but are probably like really important. I'm about to learn some math tonight. Well guys, thanks for joining. we'll talk to you about subscription limits and token efficiency another time.
49:59 >> Sounds good. Thanks for having us. >> See you guys. >> See you now.