Transcript
0:00 You know how text is just a sequence of letters? And LLMs learned really well how to predict the next, and then they can complete the text. Human actions are very similar, but in our case, more importantly, companies actions are similar. You can represent a company as a sequence of actions. And then what happens is that you can create a digital twin of the company.
0:35 What are the hardest problems that you're throwing at Fable? That you're throwing a Claude? Where you've been surprised by success? I came up with a solution to a problem that for seven years I've been trying to solve. And I was unable to solve it, and LLMs were not able to solve it. Something called the Wasserstein loss function. Making everything Gaussian. It's not that LLMs take away the drudgery of work or the work that you didn’t want to do.
1:03 And now they do it. And so now you can do interesting things, more sophisticated. It’s that you can do things that you previously couldn't. Like, in no amount of time, no amount of helpers, I could have had thousands of the best mathematicians at my disposal, I still wouldn't be able to do it. But with the model I can. And for the best problems right now what I see is that I cannot solve it without the model, but the model cannot solve it without me either.
1:32 It requires, this back and forth and creating that centaur. You know, using chess analogy, that together is better than the model and better than a human being. So you're raising the floor. But actually, don't forget, you can also raise the ceiling. Exactly. That's that is my point. Yeah. Raising the ceiling is is much more impactful. Because raising the floor, you could have just been more diligent or throw more people at it. Ceiling is, there's no easy way to extend.
2:04 Right. That's a real unlock. So would you do this? I guess, for, you know, like, research or some problem that hasn't been solved before? You can think about it that way. This is a novel problem. The solution was out of reach. Now we've solved it. In a company context, in a business product context, how do you think about the ROI or something like this? Because something I hear a lot is when you raise the floor, it's really easy to measure because you're like, okay, this amount of work would have been done by engineers.
2:32 Now, Claude can do it. Engineers are freed up. That's fairly easy to quantify. How do you think about the other kind of problem where you've raised the capability ceiling and now engineers are doing something they couldn't have before? It is a very good question. And I feel like in terms of AI, I'm just going to go ahead and say it, it’s probably the best in the world right now at measuring how high is the floor? They have not only, of course, just like everybody else, estimate the number of PRs and complexity of PRs, but we, we actually do things like estimating complexity of the project.
3:11 And in our AI system, just the system get stuff done. We could we could see that, hey, the teams of people are able to execute on more projects faster and, normalized for complexity. You can be very precise at measuring percentages of productivity you're getting. How do you have this conversation with, with Toby and, what's your advice to, another CTO at a different company that has to have the conversation with their CEO about this?
3:41 always use the largest model because the largest model raises the ceiling the most first of all, you don't know unless you use the large model and then and compare the results. You don't know what you're missing, right? It's a no no, no. And that and that they think stay in second. Always throw maximum a thing at the problem. And then until I see the evidence. Otherwise that's what we're going to be doing. And that's why, at Shopify we have unlimited tokens, for everybody.
4:11 That's why run, the heaviest models. We can find, that's why we, we throw a lot of, a lot of resources in our with, GPU inference and training and ability to optimize. In a way, it's a it's, you just have to you have to make sure that everyone that's making the decisions is actually using the model and using the product. They understand what this is. And then you have to just keep pushing the ceiling and seeing those returns.
4:37 And then it's just obvious if it works or it doesn't work. What's an example of one of these projects you ship that that raised the ceiling in an obvious way. My the one that I'm probably, most proud of is is our HST prediction system. The idea is basically, How text is just a sequence of letters. And LLMs learned really well how to predict the next, and then they can complete the text. Human actions are very similar, but in our case, more importantly, companies actions are similar.
5:07 You can represent a company as a sequence of actions. The company is, I don't know, open credit line ship product. Click add and make changes. They start accepting credit cards, take a loan. And I mean these are large changes. But all the myriad things that the company, all of its employees, every second are doing, it can only be turned into a sequence of actions. And then what happens is that you can create a digital twin of the company, and then you can start doing things to it instead of the real company and then see what happens.
5:41 You can have, counterfactual interventions. You could say, oh, what if we added a advertising campaign? Or what if they started shipping, one day faster, or what if, it's all about our merchants, what if we give them a loan and then we find a sequence of interactions that would maximize their chances of growth? And then we actually do that in production. And so we go offer them a loan or we, push them, send them communication.
6:11 Look for folks. Your most important thing is to, to start shipping faster or with, start advertising campaign on Twitter or, So, having ability to play with a company without endangering anything, real people is is huge unlock. Because now now you can you can help more people to to be successful. This is very clear. Log. That wouldn't have happened. And that has very direct impact on our bottom line.
6:43 And then the financial result. And just building this, let's say you could have built this, how long would this have taken in the past But this is the thing I don't think it was possible in the past. Right. Just think about it like you would need to get the concentration of talent. You would have to bring that. And then and then you have to have have this premonition that you need to do that, and then you need to start getting collecting data and all this, I don't think, I don't think that was humanly feasible previously.
7:13 and so the trick here is the answer is like previously, did they continue to we just wouldn't have done it. And that's the if my definition of raising the ceiling. I think you start doing something that previously was not even in consideration set. What's something that you changed your mind about because of because of Claude. Because of the model. I thought that the lens cannot really help me manage the team better. I could not have been more wrong.
7:43 And after a while, I realized, I, I flipped so much, I created a system. That actually analyzes what everything is happening like. and proactively tells me, But this it looks like this project is going to start slipping there. Deadline. it allows me to detect issues before they actually happen and go over it and then detect people unhappiness and so the whole system starts functioning much better. Has the role of it. Yeah. But, Shopify changed. now technical side is even more important because the simple things I learned will go correct.
8:18 But, the LLM is very good at picking up signals and it tries to please. And so whatever you ask, the answer is very often in the question. And it will just elucidate that. And they give you exactly what you want to trade. It's really important to want what's actually needed not put in words first thing that comes to mind. Because then then your consequences could be actually worse than before, because before, at least it wouldn't be immediately built.
8:44 There would be some discussion. And now it's so much easier to just go and build it. Technical acumen, becomes more important. And the human component of, supervising now, engineers who are supervising LLMs rather than supervising engineers who are typing, pressing buttons, that's a very different thing. That's it's almost like you get second, become second level manager, because now you want to share. Hey, what the best, best way is to dispatch the work to different alarms? And how are you going to not forget what they're doing and then how are you going to avoid conflicts between them?
9:18 I think, that's why I wouldn't say it's dramatically different in, in that sense, but I say the accent is became much more on those two things, like keeping the team more efficient and making them more have making technical decisions, and less on checking day to day. Hey, because that thing been done, Okay, I want to end on this. What is your advice to other CTOs, other technical leaders that are also trying to figure out how do I adopt quad?
9:51 How do I use it? How do I scale it out to my org? How do I get my work proficient? my main advice would be try to not limit tokens you always need, like circuit breakers, because they're all these runaway processes that suddenly start burning tokens and and, doing things bad. But it might not look like there's big difference between big models and small models. There's a world of difference and you want it. You don't know the counterfactual, so you want to know which one, which one is good.
10:22 Always use this dichotomy barbell strategy for anything coding for anything. Development, testing, auto research use. The largest model budget for it is that cost for everything production use, fine tuning is the right model for that production. Auto research the hell out of it so that you balance both, cost, throughput and quality. And save money on production by using cheaper models, often fine tuned, and invest that money in development site on the on the place where there's huge potential in upgrading your model to, in Fable 5, I was on the record to say that it's it was such a huge step, like much, much bigger than, like all the previous steps.
11:13 Yes, I've seen repeatedly people doing the opposite and trying to save money on, coding tokens on unquote on development and then at the same time wastefully running, more of those larger than they need in the actual production services. I feel like I will outcompete anybody who is not on that stretch. I was chatting with, my friends here that, Boris destroyed my, my will to live freely in my, I kept, loudly proclaiming that I will end up, deciphering, Linear A, which is a very famous, writing was used on Crete by Minoans, in the Bronze Age, kind of personal area of interest of mine.
11:55 And then, and then Boris tweeted that somebody did it using Claude, and I'm like, damn it, that was my whole spiel. I took my retirement project. I was going to work on. You had like, your old Commodore I just didn't finish a time. Exactly, exactly. Thanks so much for stopping by.