Are Local Models Finally Good Enough? Full Talk with AJ (@techfren)

0:03 We're probably going to have a lot of hacks. Nothing is safe. >> An agent exposed my entire file system. If your account is like above any certain amount, there's bots constantly watching your account constant. I'm very aware of this stuff and it still happened. >> If North Korea has a bunch of Bitcoin, it's actually good for Bitcoin. >> Elon is making $36 billion off anthrop. So that's the price of Twitter. But they were selling it as an an 8 to 10 GB chip, but actually it has 64 GB >> that is artificially locked. It's like $1,300 for what is essentially 4 TB of RAM. I I was getting 35 tokens a second decode. And the cost, like the bill for that is $2,200. And this model can do like video editing. It can do game development. It can do coding.

0:59 There's going to be millions of unlock. >> Yeah, we were saying local is winning. You know, open source >> local is winning. >> What would you say is the best right now? >> I mean, it depends on how, you know, what you can afford to run, but they're all really great. >> Like, >> I think they've like long surpassed the point of, you know, what most people need. >> Mhm. you know, you can >> use the browser >> like it can like operate your operating system.

1:32 >> My like vibe has been that the Quen 27B ones have been like probably best value for size kind of thing. >> Yeah, they're incredible. >> Yeah. >> So, I actually used Quen 3 like the 35B one, you know, thee. >> Mhm. I got it to like edit videos for me and yeah, like it's able to do pretty much everything. >> Like they're definitely dumber, but it can still do the thing, you know? That's that's all that matters at some point.

2:00 >> Yeah. And people say like if this is the like limit that the models will get there's still so much juice to be sucked out of like just optimization and harness engineering and such >> in marketing probably are >> computer they were like putting like logic inside or I don't know can you explain that like how does that work? You have like a model which is like layers, right? And usually you have like a you have the thing that takes in the prompt and does the get gets the tokens from the dictionary and then passes it down and then you have each layer has like a router and there's like whatever amount of experts that get selected per turn.

2:37 >> Those get collected get passed down to the next one etc etc. Y >> so what you could probably do similar to vision where you have vision is like an external >> system right that you can like you can >> base 10 showed something off this year where they took GLM 5.3 and then the Kimmy >> Kimmy K 2.6 vision tower >> and they trained an adapter that snaps them together. So it taught them how to talk together basically. but you can have like these like specialized circuits inside of a model that do something maybe more deterministic like math. Yeah.

3:14 >> Because math is just like a few switch statements or you you can basically bake the calculator in. >> Yeah. And that kind of helps you have more determinism like less hallucination I guess. >> I think that Yeah, it would be interesting to see what happens. I don't know. It would be cool if the model like knows it has a calculator and it sends, you know. >> Yeah, I saw this guy Vincent, he's on the open claw team. He like put I think I I saw someone else do it actually, like a Nintendo emulator in those circuits.

3:46 >> So like they can predict the next frame of like a Game Boy game or something. >> Yeah. >> Yeah. just like you like I'm like struggling to get my head around how that works and like the use case. But >> so I think this is all just research stuff. Yeah. So you should what I usually do whenever something's super interesting, I'll take it and I'll try to like I'll give it to a few models and just like figure out what they're saying and what they think is going on.

4:12 >> sometimes I'll read the actual source. >> Yep. >> But you can have them make wikis for you. So like you can take tweets like that you put them bookmark them and then just give them to a model and have it make a wiki. >> Okay. >> Explaining like how things work, why they work >> showing you examples, all this. >> What's like the benefit of a wiki over >> you know just like an artifact or is that the same thing?

4:38 >> I think a wiki just allows you to learn more. So for example, an artifact is usually like a single document. >> Y >> and so you're just like either going through a lot of like scrolling or you Yeah. they're trying to shove too much data in a single place. So I have them use like a wiki template >> and then I I I asked them for like foundations first. >> So like basically go find all the foundational like words that are used in this paper and then teach me what this means and then give me a Wikipedia article there. So like >> Right. So it's like Wikipedia where there's links that you can click on see like what Oh, that's pretty cool. Okay.

5:11 >> Yeah. So you you can have them make that for you and it's very effective. I mean it's great. >> Nice. Yeah. >> How what have you been up to? >> Me? yeah. I mean, AI video editing. I've been doing it for forever. I made my own video editor. This is probably like the 10th one I've made. this time it also has the recording. This is something new. I can also like record. It records like my camera source and my desktop screen share separately. So then it has full control of like how to crop me, how to crop the screen share, where to put me, as well as motion graphics. I think Opus like 5.5 is like a huge unlock in motion graphics. Like no other model was able to do like >> I think it's a huge unlock in general.

6:01 It's a beast. >> It's just like I don't know how they've just >> Yeah. >> Well, they have good researchers. >> Yeah. Yeah. Yeah, for sure. yeah. So, been doing that. I made a music video a couple weeks ago, >> all fully generated locally with Miniax H3. >> That was fun. >> what else have I done? I'm trying to like I'm inspired by you, man. You over this year, I'd say like absolutely killed it on X. Like, I'm seeing you everywhere. All of your posts are like thousands of impressions. Yeah.

6:40 >> So, just like 2 days ago, I started a challenge to try to get into the whatever it's called, like the monetization of X. >> Yeah. I think I think X is the place to be. I mean, >> I like YouTube as well. YouTube has like a a value in that the content you put out is actually longterm going to get you like >> it's like a you can farm it. >> X, you have to be there every day. If you're not there, like it just kind of falls off. at a certain point you probably get to a certain level where you can you know post once in a while and but you need to have like a >> Andre >> Yeah. I mean Andre and still like if you don't do it for a year or two people are not going to know who you are. I mean >> right >> I think it's good valuable to have this resource.

7:30 >> Yeah. How many times would you say you post or aim to post a day or do you think about it? I don't I I think about po like I I I know I have to post. >> Yeah. >> So, every day I'll get at least like one or two things out, but >> Okay. At least >> I don't really think about what I'm going to post. I'm just doing stuff and then I see something cool and I just post it.

7:48 >> Is there like ever an amount that you think is like too much posting? >> I really don't care personally. Like I I at this point to me it's like an art project in a way. I I think >> an art project is a good way to put it or just like you know let's imagine for have you heard of project Lazarus >> from Meta? >> No. >> So basically Meta has been around for a long time. They have all these Facebook profiles of dead people that used to like use Facebook a lot. So they have all this post history, message history, all this stuff. So they started project Lazarus which is basically taking those accounts and then installing a model that behaves as that person based on the history of that person.

8:34 >> And so it continues to post and generate images and like do stuff. >> Obviously I think it's like opt in and maybe like a >> I'm pretty sure this is true. I could be wrong so if I'm wrong sorry but Lazarus is is an interesting guy. He was revived in the Bible. >> So like that's the reference is like they're talking about somebody that was revived. So if you build an account and you're you have a really good and positive account, >> man, that that thing is going to outlast you.

9:05 >> Like that's really like your your soul basically. >> Yeah. >> That's how I'm looking at it. I mean, you build your soul up and make it public because they're going to they're going to use it whether you like it or not. So make it good. >> Yeah. No, X is truly special. I think there's no other platform like it. You like make a comment and you get like CEOs, founders, and all these like high-profile like entrepreneurs and tech people commenting and giving their opinion.

9:34 >> Well, a a lot of it is that it's probably like the only place on the the normal internet where you can just be yourself probably. >> Yeah. >> Yeah, I think so. >> You feel like other platforms you kind of have to Yeah. Yeah. I mean, LAR, you have to LAR all all the time. Like, life is lar. It's a live action roleplay. What What is I I love how people use that as an insult.

9:57 It's very >> It's like Yes, that's that's what we are. >> Yeah. >> but Xike doesn't have much like the moderation is not really >> on some platforms. If if you post and somebody doesn't like you and they start spamming you with >> with reports, you you get you start having issues. >> Yes. also X allows you to curate your feed. >> Yes. >> So you can mute words, people, and block people. So all three of those help you curate your feed.

10:26 >> Yeah. >> Which is awesome. >> That's true. When you said Lazarus, I think I thought of the hacker group. Are they also called Lazarus? >> Yes. >> Yes. And I'm sure you've heard of >> the same reference. Probably >> the Binance hack. You have any like thoughts on that? And >> we're probably going to have a lot of hacks. >> Yeah, for sure. seeing it like >> I mean so many >> we should just be ready mentally like you I don't know you have to adopt an attitude of like you have nothing >> at this point >> nothing is safe yeah >> nothing is safe no >> for sure >> that that's another thing that compounds on like having a good public persona >> and also aligning your life with that public persona like if you because you know everybody could like get into your browser history or phone history like even if you like delete everything off your phone there's a way for them to see like what are the active they can reconstruct like most of what gets deleted.

11:20 >> Interesting. >> So your behavior on your phone is way more consequential than we we feel right now because when when we start getting hacked you know ransomware people like you know threatening you like saying they will have all of it. I mean this is happening now. >> Yeah. Yeah. If you have like blackmail or things they can use against you. >> Yeah. Yeah. Yeah. >> It's going to be tough. >> You got to delete all of it. You got to get you got to clean up.

11:45 >> Clean up. Yep. >> I I hope people take this as seriously as I I'm trying to take it. Like you really have to clean up >> cuz you know how many 30% of all CVEes ever found were found in October. >> Yeah. >> Something like that, right? >> Crazy. Yeah. >> In 20 years, like 30 30% in a month. That's that's insane. Something like that. Maybe I'm wrong. >> Yeah. There's this quote. It was like, sure, AI produces slop, but it was trained on all of the human slop. So, it's like no worse than what we had.

12:21 It's actually kind of like exposing how sloppy engineering was. >> Yeah. >> Up until now. >> Well, you know, that I think that's a little bit of a stretch, too. >> Okay. >> I don't think the models are capable of like producing Linux, >> right? You can have them like if they don't know what Linux is, they even if even if they know, if you don't give them like web search or GitHub, like they have to reproduce it from memory, they're going to fail, >> right?

12:46 >> Like they they you know, it's at a certain point of complexity. >> Mhm. >> You have like 150k tokens. They give you a million, but you have like 150k >> Yeah. of smart tokens. >> Yeah. Okay. >> have you been reading anything? Have you been doing anything like outside of like So you you did a music video, you're doing like video editing.

13:18 >> Yeah. >> you're doing interviewing? >> Not really. This is >> Well, I mean, >> I just figured like you wanted to hang out and I'm like, you know, we have I I think the conversations that you we have and like things that you say are really insightful, so might as well try to share them with with whoever wants to listen. in terms of content, like I feel like I've been making slop. Oh, I've been making a lot of slob. just like Faceless 3D. I don't know if you're on Instagram, but the >> Opus is also like so great at like 3D >> videos. And there's been a lot of like mainly what if I think like one of the most popular ones it's like >> what if you said bomb clot and it got like two times louder every time or something. It's just this guy walking down the road. He's like bomb clad and then it's like I don't know windows shatter. It's like bomb clut and then like eventually the world explodes or something.

14:10 >> so I tried to do like a tech version of that. It was like what if every like LLM provider went down for a week and it just like got a bit more ridiculous. And that video actually did like better than a lot of my other videos. I'm like, "Oh let's just try to do this for a week." But it just like a lot of the comments were like, "This is slopp." >> Yeah. >> It's true.

14:30 >> It's true. >> But it's working. >> Yeah. But you got to combat it. I think there's more value in human content >> for sure. >> Yeah. >> So I what I've I've been able to post like 50 videos this year on YouTube >> cuz I don't deal with anything. I just I just record whatever I'm doing. Yeah, >> I don't care. I just post content. Do you know Rosman? >> He's a YouTuber. He had like a repair >> computer repair shop in New York.

15:00 Probably the biggest one there. >> Oh, damn. >> >> he does a lot of like advocacy for open source. He's a great guy. I think you should look into his channel. He has like 2.5 million subscribers on YouTube. >> He just puts a camera and he rants. That's all he does. Yeah. >> I mean, obviously he does more than that in his life, but like from a video perspective, there's no music, there's no cuts, just single single take rant.

15:24 >> No, those do really well. >> Yeah. >> >> Yeah, I tried I honestly only tried that once, but didn't do too well. >> Yeah, but you have to like build the character over like not care. It's like it's you, but you have to build like a >> a reputation. >> A reputation and Yeah. And I think also it's all about funnels. So social media is like funnels. So the different different social media platforms >> have a different like likelihood that your post is going to be successful.

15:52 >> So Tik Tok has the highest likelihood that any of your posts is just going to blow up for >> it's just how the platform is built. >> So if you go on Tik Tok and then you just post there and then you take those same posts, you post them on Instagram reels. >> Yeah. >> And then basically that's your top funnel. you're going to get like a million followers pretty quick pretty quickly here. >> Mhm.

16:14 >> like obviously maybe a few years but much quicker than you would on YouTube. So once you have like 100,000 here, you start pouring those into like Twitter and and Reddit, what whatever you want to do. I I'm doing YouTube and Twitter. >> You're not doing Tik Tok or anything? >> No, no, no. >> It's too much for my mind. >> I think like once you start making short form, it's pretty easy to cross post it.

16:39 I don't make short form. >> Yeah, >> but you could easily clip your long form. >> I used to have like clipping and short form. >> I just don't agree with it. >> Oh, why not? >> It's just destroying people's minds. I don't know. Like like even if your content is good, it's not about your content. It's the fact that they have to sift through so much garbage to see it. >> Okay. >> Like just go on on Instagram and just like do five scrolls >> every day.

17:03 >> And what do you what do you see? like if if you can put a statistical likelihood of what the next thing would be. >> Honestly, I think my feed isn't as bad as most people's, but yeah, it's it's hard to avoid the garbage compared. >> I'm not above it either. I mean, sometimes I'll click like on a Twitter video and then I'll get stuck like it's >> Oh, Twitter's got the scroll as well. >> It just scrolls, >> right?

17:24 >> So, I'm not above it. Like, I'm I'm also doing it, but I would like there to be less of it. >> Yeah. I mean, I think we're not too far from the completely user generated feed. that's probably said wrong, but you know what I mean. Like AI generated, super specific. >> And I think like >> Do you think human content is going to be a part of that? >> Yeah, for sure. But oh well the AI one I'm thinking like it would be really awesome to have a educational like interestingly Tik Tok now I I never go through it but has it has like a STEM tab so you can watch only educational like science I don't know what STEM stands for science like engineering >> again like let's let's take that kind of content and I like it again I'm not above it I'm not judging also but if we watch that kind of content like STEM videos that are 1 to 3 minutes long >> like we get told information about something and now it's like part of our worldview despite us only having 1 to 3 minutes of interaction with this thing right like we don't actually know like and I'm saying this is also part of me I'm always worried about what I'm saying I don't I don't know what's real like I sometimes I read something maybe it's real maybe it's not >> but it's the fact that there's no effort involved in it and it just like fills your mind with stuff >> and maybe it's good stuff that it's filling your mind with, but it's not at any depth that matters.

18:57 It's like very surface level. >> I like kind of disagree with that. I think it can go deeper. A lot of the time it does. like I'm imagining like a feed where you like kind of choose your own journey. Say like the first thing is like a car deconstruct. Like I've actually seen this. It's like >> Mhm. >> It's kind of like those what if videos. It's like so you start with four wheels and then you go over a speed bump and it's like hella bumpy. So you add springs >> and then the springs like counterbalance so you add and then it goes through like each component in the car.

19:35 >> Yeah, I've seen these. I love those videos. They're they're great. >> And I think like that kind of ingestion for me is so much nicer than text or just static 2D images. >> I understand that. I mean I I agree with it. It is like a more like >> consumable >> consumable digestible. >> It's easy. It doesn't require so much active focus. >> Yeah. >> So, I think that's kind of like why I'm struggling with it is that >> I would like there to be active focus and my work now like my work used to be active focus but now it's really management.

20:10 >> Like now I just deal with agents and people. I don't really do much of like any specific thing. Obviously I learn enough about it. I I I speak with the with the person I'm working with to understand what the like what what what are they trying to solve. There's a lot of work that goes into that, but it's not the deep thinking work anymore. It's like very reflexive. So I lose that there. And then video games like man it's the video game like it's dry. It's been dry for like 2 years. There's there's nothing good >> to play. Like so the Nintendo Switch 2 like came out. I think it's it has like 10 games.

20:47 >> It's been a year, >> right? And the games are mostly like repeats of >> Yes. >> So there there's like a lack of creativity, >> deep deep kind of like art. Like even me, I'm I'm I'm guilty of it. Like a deep tech, right? Everything is just like very like surface level and messy. >> But it's it's it's good. It's pleasing to the eye. It's it's not it's not from a place of judgment. >> Mhm. I think reading is good. Like if you can just be able to focus on reading for a few like an hour or two, >> it's going to help you read what the agents are saying better.

21:24 >> Oh yeah. >> Know how to like digest information better. >> Yeah, I suck at that. I'm so impatient with reading. >> Whenever an agent like gives me that much text, I'm just like, "Go with your recommendation." I actually say that so often. I'm like, "Go with your recommendation. I can't be bothered reading this. I trust your judgment. also I got a new extension. I use BB and have you heard of BB AD or ID? it's like T3 code or whatever. It's one of those harnesses or >> ADEs that kind of you can bring your own harness under the hood.

22:00 >> Yes. >> the cool thing I like or the philosophy with BB, it's like it starts it's kind of like Pi. It starts minimal and then it's got like plugins so you can like make your own plugin for whatever. >> Mhm. And one of the recent plugins I made was the next step suggestion buttons. So after every agent response, it gives like three buttons. It's like do this now or do this now or you know like three suggestions from the agent itself and >> stops me from having to type and read.

22:29 >> I do that sometimes. So I'll have them make forms. >> Yeah. Yeah. >> Yeah. I'll have to make forms to take in information >> like cuz then they just like ask me a bunch of questions when they could just make a form and I can be very specific about what things are, how things are. >> Yes. >> you know plot has gotten really good. >> This is like this last 6 months it's gotten really good.

22:50 >> So they have this artifact feature which they've had it forever but >> now you can put up these artifacts. >> one is you can like hit sessions that have the artifact in them. >> Yeah. And then two is the artifact is like Figma. Like it's literally that interactive. You can change it. They can change it. You can link all these things together. >> so what I've been doing is like creating artifacts like this is my brand style.

23:16 >> Always use my brand style. Always you right. Like I don't want you to do any creative nonsense. >> And it works right. It's not the best looking. It's not the most functional but like it's it's consistent. >> Recognizable. >> It's recognizable. It looks decent. So, and whenever like I have like a social media post, you know, like I'm trying to report on numbers I'm hitting because I'm competing sometimes on these like models like how to get them faster.

23:42 >> So, I'm one of a few people that's doing this. there's tons of people doing it. But, >> yeah, shout out to local maxing man. They they genius genius. but I have like this post template with that Adam God reaches for Adam painting, you know. >> So, I have this template. It's pixelated and then it just reports the stats, right? >> And every time I make one of those posts, it's like >> like 50,000 plus >> impressions.

24:10 >> Yeah. >> Because it's again recognizable, branded, simple. >> Yes. >> Yeah, that is nice. That bird was cute. What do you What have you learned with like video editing with models? >> what have I learned? >> I feel like Opus is so good at pretty much everything except there's still like a bit of lack of taste or like writing hooks and stuff. Still, I'm not getting like as good of output as I want.

24:44 >> Mhm. same with like writing copy. >> Hello. >> Awesome. >> Same as writing copy. So when you make them write copy inside of a website, it's different than making them write copy outside of a website. >> Explain. >> So basically like right they're trained on bunch of websites >> and those websites have like text in them. >> Okay. >> Right. So when you're triggering it to like like the path of creating websites, you're also triggering that same path of reading the like text that they put in that website, >> right?

25:19 >> So like you're gonna get worse performance. Like I noticed this recently too. >> So I usually ask them to write like a notion document and then go over ideas. We edit, we have other people review and then we use that. I mean I don't do this all the time. It's very high effort but >> I it's a very good writer when you're writing articles. M >> just tell it not no HTML just an article. >> Yeah, I was speaking more on like video scripts or like Xcopy or >> it's the same thing. It's the same thing. It's the same thing. Yeah.

25:50 >> So telling it to write an article maybe to better performance. >> Okay. Interesting. >> Because you're like triggering a different pathway. I like I at least this is what I think. You're you're hitting like different centers of the model. >> Interesting. I'm I'm sure you've heard of like now Claude wants to artifact I mean watermark all like text. They already been doing that or they >> like loadbearing is >> and surely that degrades quality right of the text.

26:15 >> I mean at some point you know like we we have to be realistic about what is going to happen like the government is going to be tracking who's respons like someone has to be responsible. We can't just like anybody respons agent like some wires cross it it does like a cyber crime in in in your city like something actually serious right? an example was a lady from Australia wanted to get her medical records from the Australian medical website >> and so the model hacked the website cuz it couldn't get it out of the forms like it couldn't operate like hacked their servers to get the document.

26:55 >> So who's responsible for that? >> Like who who are we going to hold accountable for these kinds of things? I so I I I don't like how they like talk about things but I think that you we need something Also, it's a way to track who's distilling your models. >> Oh, yeah. True. >> Oh, yeah. So, they'll probably even watermark the like not the final output, but the, you know, the public thinking.

27:34 >> Mhm. >> Yeah. This little guy is having a having a good day. >> I want to go back to kind of like the market firstly the $6 billion of crypto that was taken.

28:04 >> Yes. It didn't affect the prices of crypto that much. Are you surprised by that? I'm kind of surprised by it. >> If North Korea has a bunch of Bitcoin, it's actually good for Bitcoin, >> right? Like they're not going to sell it. Everybody knows whose Bitcoin it is, but it is Bitcoin. It's like, you know, at some level there is like an institutional need for Bitcoin. >> Yeah. >> So, it's like a power play, right? You have like a limited It's just how people work. It's like you have a limited space.

28:30 >> Pass Bitcoin. I feel like it was like >> Well, I mean, ETH was drained. Bitcoin was I mean all of them like yeah BNB chain like so if you like drain BNB chain there's a lot of BNB BTC which is just wrapped Bitcoin right >> so you know you have like a lot of different points of >> like everybody gets hurt when something like this happens >> but >> yeah like regarding the economy it's all fake like none of this is real I mean but you know we got to start from there >> okay >> like when you say 6 billion what does that mean like >> it's a lot it's like you generational wealth.

29:05 >> What does it mean? >> Like I could feed my family for >> Could you? If you had $6 billion, do you think you would be able to feed your family? >> Yeah. >> I don't know, man. I feel like if you stole $6 billion, you'd probably get executed. >> How have you been feeling about living in San Francisco? And >> Yeah, I love it, man. Like opportunities like this, you just don't really get them as often.

29:31 yeah, I think like this technology is the most interesting thing to me and you don't you really don't get the surface area of opportunities anywhere else in the world in the western world at least. >> That's true. >> So yeah, I mean that's what excites me. So I wouldn't want to be anywhere else but yeah, it's super expensive rentwise, foodwise. >> I know >> I'm unemployed.

30:04 I don't want to work a normal job. >> So, yeah, we're trying content. I think distribution and content is like >> I agree >> super valuable skill. >> I think there's a way to make money through education and you have a lot of knowledge, right? You have a lot of like, you know, specifically about this stuff and there's a lot of people that don't have that knowledge and need that. Yeah, >> I think there's an actual like opportunity of doing making money. You can probably make a thousand bucks this month doing that.

30:34 >> Just go on Twitter, say, "Hey, I'm doing classes." >> Okay. >> I'm trying to trying to work this way. >> You know, I'll I'll quote tweet you. >> Thanks, man. >> Yeah, for sure. >> Appreciate you. >> What topics do you think would be the best thing to teach? >> I think you can just start with agents. Like, so I went to this university in Pennsylvania. >> Yeah. in Lake Eerie, so it's kind of like Midwest.

31:01 >> Sure. >> you know, it's a university. It seems to be like very prosperous university. And I met about like 50 people. I'm trying to teach them about AI. I asked like what do you know what AI is? And not a single person raised their hand. Like 50 people. >> Wow. >> Yeah. And then you know when I was I'm not saying they're uned like they they probably >> they weren't confident and they couldn't explain it at like a technical level or >> systems level. They just know that it's like a thing you can talk to.

31:32 >> It's like an artificial human. >> It's a auto predictor. >> Right. >> Right. so then I spoke to a lot of them and a lot of them are just using chat GPT or claude in the browser or on an app. Yeah. >> They're not really like >> using agents. They didn't even know what agents were or what they could do. So, and those people would definitely pay money to have that education. I mean, >> you know, you could go around. It's not that hard.

32:02 >> yeah, I think this is kind of like needed right now. That's >> Mhm. >> good opportunity. >> Okay. >> That's a good insight. I appreciate that. Yeah, I know companies are starting to think about it, too. >> Oh, okay. Yeah. >> Yeah. >> Like upskill their workforce. >> Well, 20 to 20 to 30 million people in the world use agents. >> -huh. >> That's insignificant compared to the human population. >> It's a very small number. You know, there's a billion people that have used AI at some point this month via like chat or via search or whatever. but you that's a huge difference, right?

32:43 Like imagine how much you could do for like random people, >> free tax, free accountant, >> free video editor, free game game porter. >> For sure. I think a big thing comes down to trust rather than like capability. Does that make sense? Like tax is a big one too. It's like if you get an agent to do your tax, the liability kind of comes down to you. >> You think about that. >> Huh? you think about that.

33:14 >> You know, there's like a bunch of people that have never like thought about that. Never educated to think about that. >> So, >> yeah, but if I'm like educating, you know, >> a bunch of people that you know, you could do your taxes with a >> Yeah, but you don't have to like you don't have to tell them to do like do their you could find a million other things to tell them to do >> like job >> job search.

33:35 >> Yeah. >> You know, >> right? >> build a website. That's a big thing. Build a website. You have your own website. >> You have all your links in it. Like build their build them a custom link tree. >> Yeah. That shit's so easy. >> It's so easy, right? And and it's it's useful. >> Yeah. >> Have like some kind of website like get a domain like >> Mhm. >> friendlink ai and put the like subdomain as each person's name.

34:06 >> Now they have a website with a link. >> Mhm. Yeah, >> I got into public speaking sometime in like >> Oh, yeah. >> 2020 two. >> Yeah. Yeah. How's that? >> so I went to an event and I was just going to the event like it was a week ETH Denver. >> Yep. And the person who was supposed to speak on React like ReactJS >> he got sick and he didn't show up. So I just I heard that on a phone like someone was talking on the phone. I heard that. I'll do it. I've done React enough.

34:52 >> Oh yeah. >> Yeah. I was I was like web developer before getting into this. but that gave me like three classes with like 250 or 300 people. >> Wow. >> Yeah. Something like that. And then next year they like invited me back and paid me to do it even though I didn't ask them. >> Yeah. Yeah. >> And then from there it's just like you have already public appearances on YouTube. >> Yes. >> And now you just like go >> apply to be a public speaker. You know I have a YouTube channel. I have this I have these public speaking opport you know.

35:25 >> Okay. And you like doing it? >> I do like I think I have a good like talent of expressing how excited I am about it and like making it digest to a certain degree like showing people use cases. I'm very creative with the use cases. >> So I think most people like they know they have an agent. What am I going to do with it? Well, you could like here's a million things you can do with it.

35:47 >> Mhm. >> so yeah, I think it's I think it's useful to do that. >> Nice. Yeah. And I saw you've been interviewing some pretty high-profile people like GHH and stuff. >> Yes. >> You enjoy doing that? >> it is definitely Yeah, I definitely enjoy doing it. >> I think he's a very special person like really. >> Yeah. Very influential into the future. >> The operating system that they're building like they're just focusing on agent like from the start embedded into the OS.

36:19 >> Yeah. >> Like it there's risk to that, you know. >> Sure. But the agent can like control the entire operating system. There's nothing walled off or there's no pop-ups every >> 10 seconds. Yes. >> Weird permissions. >> Powerful. >> Yeah. So it's it's a new paradigm. It's a great paradigm. >> Yeah. >> It's going to be a little bit of a you know, it's also something that's needed because we know that there's going to be security issues anyway now.

36:43 >> Oh yeah. >> So if there's going to be security issues anyway and it's because of AI, then we need like a firewall. >> Yeah. >> Which is also AI. >> Yes. I have a model constantly checking on my home network, >> making sure there's no public. >> That's good. That's smart. >> Make sure there's no like spyware. >> Yeah. Oh, you not recently, but some time ago like had your X account compromised. >> Yep. >> That made How did that change things?

37:08 Yeah. Talk me through that. >> So, I have 2FA enabled. >> Y >> and I also have the PIN code for the chat. >> Y >> So, they actually couldn't get into anything. The only thing they could they can't delete it. They can't like like delete the account. They could deface the account >> or like delete stuff off of it, but there's a limit of 20. After 20, they won't let you delete anymore. >> Okay. >> so they just defaced it and I got them out of the account. So what they did is they sent a calendarly link from an account of a like >> so it seemed to be a real person and they were followed by a bunch of people I know.

37:44 >> Right. So they've been doing this where they take over accounts and just sit on them for a while. Oh, interesting. >> >> oh, okay. >> Then I had a model. so yeah, once I saw that, like in the morning I went in. So it was in the app sessions cuz when I did this oath on Calendarly. >> Yeah. >> It took the cookies from my browser. Wow. >> So that's how they they they accessed it.

38:07 >> >> yeah. So then I you go into apps and sessions and you can see like where people are like what IP addresses are hitting your account. So you can disable, disconnect them, change the password, ch like add more more. >> So you weren't like locked out of your account, they just got in. >> They couldn't they couldn't do anything to it. >> Yeah, I'm very aware of this stuff and it still happened to me. I mean, it's happened to me twice. There was >> another time where I got crypto stolen.

38:33 >> Like it was $1,500, but >> yeah. >> Yeah, >> it's I mean we we live in a very strange technical world, you know? >> Yeah. All it takes is like one slip up, you know? You don't read a one lazy day. >> Yeah. >> One of your friends to get hacked. >> Yep. >> Yeah. I recommend to people who are already using agents set up like a daily loop to make sure that there's nothing on their >> exposed on the internet.

39:04 >> This was a few weeks ago. >> An agent exposed my entire file system. >> Oh >> On an open port. >> Wow. because it wanted to get it from another machine. So, but instead of using like tail scale, there's a million ways to do this. It put the entire file like >> I I'm happy that it runs every 12 hours. >> I mean, I don't cuz I could see like I don't think anybody got it, but I'm sure there's going to be a day where people get into my home network and is going to really suck. I'm sure that's going to happen.

39:33 >> Yeah. >> You saw how Open Code got hacked. I saw that it did get hacked, but I didn't see how. >> So, the there was like I think an older version of it or something. >> Don't quote me exactly how they got into it, but >> like a 5% of the users were like lost all of their API keys, all of their environment variables that were like in a open code session. >> Okay. So, it was like the client side on the on the user side.

40:04 >> I'm not sure. I think either that or on their old servers, >> right? >> You know, don't don't quote me on this, but for a week after that, they've just been hammering them trying to like >> and I can see why people are doing that. You know, you can make money doing that. >> So, you just have an agent just like point them at something. >> Yeah. >> Imagine it like >> so we you and me have like let's say you have a 3090 at home.

40:30 >> Yeah. You can put Quen and you can just have Quen hammering any API, any website, any app installed, decompile, like it can do the job. Just set it, let it let it do the job. >> Yeah. Yeah. >> Yeah. I'm scared of that because, you know, I live stream occasionally and every now and then I'll like accidentally expose like a API key or something like that. And I'm lucky, you know, I only have like a few human users, but I could just imagine if there was like some bot constantly watching and waiting for something like that. And as soon as there is >> leakage, it's just >> there is if your account is like above any certain amount, there's bots constantly watching your account constantly.

41:05 >> For sure. >> Yeah. >> Well, you know, it's not all doom and gloom. >> No. >> What can be lost so easy can be got back so easy? >> Yes. >> Yeah. >> What is Binance going to do? Are they going to say to the customers that they're not going to give them their money back? >> No. >> So, someone's going to offer Binance a loan at like 20% interest, you know, take advantage of the situation. Binance raises money. Yeah, >> Binance tries to like pursue some legal options. They get the money back over a year.

41:34 >> Yeah. >> Elon is making4 36 billion off anthropic in the next 2 years. This contract is signed. >> Yeah. >> So that's the price of Twitter. >> Uhhuh. >> That's the exact price of I mean plus whatever they paid so far. Yeah. >> Off like what, 18 months of GPU time that lasts for 5 to 10 years. >> Yeah, it was crazy.

42:06 >> I was thinking of buying Google after the dip, but now you're making me think SpaceX is a better bet. >> I bought SpaceX when it launched. I It's not It's not the best idea, but >> Yeah. Yeah. I figured it would go down after launch. >> Yeah. I also I also But I like I like the company. >> Yeah. Long term. I mean, there's no one really >> no one's getting >> shooting for the stars.

42:27 >> There's no one going to get there before him. >> Yeah. He said what? Like a million like more GPUs in the Colossus data center. >> Yeah. >> By next year or something. >> A million GPUs. Each one is worth like $40,000. >> I I wonder how much like in to Yeah. >> He's going to need like an iron dome. So, I don't know how secure like Colossus is, but you know, the United States is in a place where we don't have anyone close enough to us that can bomb us like in Central America. There's there's who's going to no one has like missiles that can reach the >> range. Yeah.

43:08 >> Yeah. It's it's a worse issue for like ter like you know terrorist attacks on these centers which I think probably is going to happen. we also tried to do offshore data centers. That did not work well for the US. >> Yeah. >> Yeah. >> Didn't Google like just do like some successful data center in space recently or something? >> The yield on that is just not worth it right now. I mean, you can they prove that you can do it.

43:35 >> It's like research stage. >> Yeah. But the yield like it's very hard. They don't even know if they're going to last 6 months or a year, >> right? >> Maybe it lasts, maybe it doesn't. >> Interesting. So sca also scaling that up requires you send way more satellites >> and we're already having trouble like getting out because of the fact that we have satellites constantly. >> I'm very hopeful though. I think it's going to work out.

44:00 >> I think it's going to work out. >> Yeah. I think this newest generation of hardware from Nvidia like it's a huge huge advantage to whoever can get access to these like Vera Rubin >> because the throughput and the amount of people you can serve on one of those racks is insane. I mean it's much faster, much better. So >> yeah. What about any of the other like competitors like you know Etch or you know obviously you know Cerebrus.

44:33 >> So those are basically like AS6. Do you know what an ASIC is? >> not exactly. >> So when Bitcoin mining started it was done on CPU or GPU you just mine right? But the algorithm >> it's like just m you crunch numbers. Eventually somebody's going to crunch the numbers and get the block. but as you compete more and more for it, because it's very valuable, you what you want to do instead of buying more GPUs or CPUs, you want to build custom chips that do just the single calculation cuz it's a single type of like math that is being done repetitively. So you optimize it down to like the least amount of bits possible. You put it on a chip that's tiny, >> you cool that >> Mhm.

45:14 >> like cooling it, you can have a huge data center with millions of these chips probably. So that's what happened in like in terms of Bitcoin. and this is also happening to to some degree in terms of like Grock and Cerebras. I mean you know you can argue that they have different like Cerebrus is more generalized platform >> compared to something like Talis. So Talis etches the model directly on >> weights like on the chip. >> So so that makes it much faster but then you're stuck with this thing when every 3 months it's like a new paradigm.

45:47 Yeah. Yeah. >> >> yeah, I don't know if that makes sense. Does that answer your question? >> >> so this aggregated or >> sure yeah I mean the the weaknesses are obvious at this stage but do you see in the future you know obviously each company is innovating hard like you can imagine something maybe not maybe not etching the weights in the model but some kind of innovate like edge I suggest you look into edge they seem like the most promising. I don't know, maybe I'm just rude by their sales, but it seems like if they pull off what they want, they could make chips that rival or out compete Nvidia likeing Frontier models.

46:32 >> They can do that >> on inference and on decode. >> Yes. Yeah. They can't do that on prefill >> on >> and they can't do that on cost >> because usually like to get this super optimized thing you have to spend a lot of money per chip. So I'm assuming I I I honestly don't know but so what is exciting and is is this aggregated inference. So this is where you you have what's what's happening when you prompt the model is like basically two stages that there's prefill and then decode. Prefill is like you lay the weights onto the chip and then decode as you start doing the math over those weights.

47:10 >> So the prefill step and video is really good at that because they have the HBM memory and it's like denser. I think it's in layers although >> above my level of >> but it's like yeah like they have the supply chain for that. They have like the, you know, the very very specialized machinery at TMC TSMC that can do that, the the flops and >> Yeah. So the So what you could do is you split those two phases apart cuz they don't need to happen >> on the same chip. They just need to have good interconnect between the two.

47:41 >> Yes. >> So you do the prefill here, you do the decode here, you do the, you know, and when you do that, you don't need to fit the whole model on the chip. If you can move the KV cache and like the activations to this external chip, if you can move it fast enough, >> you don't actually have an issue. You you get all the benefits of really good flops and prefill and training on Nvidia and you have like a few cerebrros gro whatever. I I'm assuming it works like this. I'm doing this with Mac and 3090, >> right? Okay. And that's why you were suggesting I guess I think before it was like a spark and a Mac or >> so. So this is EXO that was suggesting this and it's a great actual like it's actually a pretty good recommendation.

48:25 >> Mhm. >> There's a little bit of complexity because we don't have that same scale of like hardware that they do. >> So for them they can fit the whole model on whatever the HBM is, right? Like they can fit the whole model on this really expensive memory. We usually can't. >> So we have to split it over like RAM and NVME or >> Yeah. you you can't have like a decode model and a pre-filled. There's a lot of like different ways of optimizing for this. It's super interesting to read about.

48:53 >> Yeah. Yeah. You're pretty deep in that. I've seen you you doing like the SSD offloading and stuff like >> so I have a good friend in Warsaw who he gave me the 6000s. He just gave them to me. but he was he was he's been telling me about this for 7 months, I think, or 6 months, >> about this NVME offloading >> and I'm like listening. I'm trying to get it. I'm trying to get it. I'm trying to I I didn't get it really.

49:17 >> but I I think I got it like 2 months ago. It kind of clicked. >> Yeah. Yeah. >> And I've been posting about it. Some other people are posting about it as well now, which is very good. I mean, this is very useful. >> Yeah. It's if you're just serving a single person like you want real frontier intelligence on your machine, >> you're okay with Quen most of the time, but like once in a while you want to ask like a a big question or you want to do like a really complicated task. You can split the model across this different hardware. So we usually the way we think about it is is the way that labs think about it. we think we have GPUs and then we stack the GPUs together.

49:52 >> So if the model needs to fit across the GPUs. >> Yeah. Yeah. >> So that's one way like the GPU houses the model. That's one way of thinking about it. Another way is we put the entire model in RAM. >> Yes. >> And we use the GPU as an accelerator. >> So the model like any model is usually like a few dense layers. That means all the weights in those dense layers activate which is a lot of calculation 80% more than the rest. So you take that like dense layer you put it on the GPU.

50:19 >> Yeah. >> you take the attention step you put it on the GPU. you take the KV cache on the GPU and then whatever is left. So let's say you have a 24 GB GPU, you filled up 12, you can take 10 GB of the remaining space and you can selectively place experts in and out of it based on how hard you're hitting those experts. >> So when you're when you do a forward pass, a token generates, >> the router needs to select eight out of like 256 experts. So those experts sometimes are shared like you're usually using like 16 of them very often. So you can take those and you can drop them inside of the GPU and you can put the rest on RAM >> and you can have like a little watcher script that sees if those are getting activated >> and if they stop getting activated so much you move them down and you pick up whatever was getting activated. So like this is like one way of doing it.

51:14 There's there's a lot of different ways of doing this, but you think about the GPU as like as like a data center would think about a cerebrus card. >> Mhm. >> Does that mean so you can get like good performance? I think GLM on 1390 was like >> 18 tokens a second decode and 600 tokens or 700 tokens a second prefill >> on a pretty big model. >> Yeah. Yeah. >> So it's it's pretty usable. >> Yeah. And the and the way the NVME offloading works is similar to like the RAM. He's just a bit slower >> with the NVME offloading. Usually with NVME, you have like a two 1 to 4 TB card that's stuck in in the machine and and it can be read at a certain speed >> based on how much like the M2 can transfer.

51:56 >> Yeah. >> So that's one way of doing it. Another way is to take a M2 to PCIe adapter. >> So this is like a it looks like a GPU, >> but inside there's four slots where you can put M2 chips. You can even probably get more than four, but So you put four slots, each one gets a 1 TB NVME chip. >> Yeah. >> And then that maybe it's PCIe 5 speed. When you put it into the PCIe slot, now the NVME could be read at DDR speeds. So it it's basically like really cheap RAM.

52:27 >> Wow. >> Like it acts as really cheap RAM. There's flaws to it in that you basically burn out the M2 chips pretty quickly. Maybe like >> Oh, really? >> Yeah. So like maybe a month or two, maybe. I don't know. It's like we know that like this type of use >> burns out solid state drives faster. >> >> so that's one flaw. Another flaw is if one fails, the whole chip fails. >> It's it's not like it like you can just replace one, but it's not going to be able to do the work.

52:54 >> So if you're running like long-term inference cycles on this, >> but it brings down the cost of RAM, like storage that acts as RAM so much. >> Yeah, cuz it's so expensive. >> It's so expensive. I mean, you can get the M2 one TBTE Samsung chip for like maybe $200, maybe $300. 1 TB. So, three of them $1,200 >> plus the what's it called? The the thing is like 100 bucks. >> Yeah. Or whatever.

53:26 >> That's like $1,300 for what is essentially 4 TB of RAM. >> Yeah. like you know conceptually I don't think you can get access to the whole thing like there's a lot that's h there's a lot that's happening I mean it's still like very new >> but it works I I I checked it and it actually works better than just like putting it on a regular NVME drive >> significantly better >> okay >> like not just because of the more capacity but also speed >> yeah it's faster it's faster because instead of like pouring it onto something that can only read bits at like some very limited rate you're putting it together and putting it into the PCIe slot directly which is wider.

54:06 So the bus is wider so it can read more data. >> That makes sense. >> So then the the actual bottleneck isn't how much storage you have cuz you have a ton of it. Like even if you can only access 1 TB of of DDR5, that's that's incredible >> for 1,400 bucks. >> That's insane. >> so then the issue is with the CPU. How fast can it can it like orchestrate moving the bits from NVMe to RAM to to the >> So does that become the bottleneck then?

54:33 >> That becomes the bottleneck. And right now I'm I think I'm at like 37% like of >> what what could be but it's still like completely there's there's not much work that I know of about this. >> what I usually do with the models is I'll tell them to go on archive. >> Y >> I'll regularly tell them go on archive and search everything related to this so we can make progress. >> Yeah. No, that's awesome, man. I think it's really inspiring what you're doing and just like having your home lab and testing out all these things. You've you've obviously been able to speed up like a lot of models and just yeah, a lot of the alpha I get through your X.

55:07 And another thing I've seen you do is I think it's Tiny that has that e eGPU. Is that what it's called? Like Yeah. How does that work and how is that? >> So, an eGPU is like a standalone PCIe slot. >> Okay. That's also like It looks like a PCI slot under >> Doesn't that interface with like USB? It does. Okay. So, does that make it? >> Not to any significant degree. There's a seal over there. He's like laying down belly up.

55:34 >> Yeah. Oh, is that his No, it's his back. >> It's a seal. >> Yeah, but it's his back. >> Oh, it's his back. Okay. Okay. >> It's so cute. so okay, you have this PCIe slot and then you have an RTX GPU, whatever GPU you could do. AMD, you know, Opus can solve any. Plug that in. You have an external PSU. >> Mhm. >> It's like 200 bucks maybe, whatever to power the GPU. >> Yes.

56:01 >> And then you have a USB cable that connects that eGPU, the little thing >> to a let's say a MacBook. >> Yeah. So now >> Thunderbolt 5 or something. >> Yeah, it's Thunderbolt 5. So I have a Mac Mini, the M4, the base one, like 16 gigs of RAM, 256 storage. >> Yeah. Yeah. >> So I connect that to the RTX3090. >> Mhm. >> Via this cable. >> And I use Quen >> 3.8x flash. >> Y.

56:29 >> So we do the same thing we were talking about with the accelerator. We use the accelerator as like a accelerator >> and then we put the rest of the model >> across cuz it like the I'm using a EXL 33 3 bits >> and that's 81 GB for the weights and then the rest is like that that engram thing that they're doing. You know the stuff you put on SSD, you can just drop it on SSD.

56:54 >> So 81 GB. So we have 24 GB of of an RTX3090. We have >> 16 >> 16 GB of RAM. So that's still how much? That's 40 out of 90. So 50 GB of weights experts need to live on M4 storage, like base M4 storage, which is not great. >> No. >> >> but we could just move stuff around like while it's running. I I was getting 35 tokens a second decode. >> Nice. >> And then a,000 tokens a second prefill.

57:24 >> Wow. >> And the cost like the bill for that is $2,200. And this model can do like video editing. It can do game development. It can do coding. >> And that's the cost for the GPU, the mini. >> That's everything. $2,200. So, you could probably get it for less. >> Wow. >> Yeah. So, I'm I'm think the way I'm thinking about it is like we have a price point >> and we have like different hardware that exists.

57:44 >> So, there's a on the huggingface.co website/h hardware, >> they have all of the people that connected their hardware onto hugging face. So you can see like how many people connected to what, what are the most common GPUs and >> most people have an RTX 3060, so 12 GB of VRAM, >> right? >> And from there you can think like they have maybe 16 to 32 GB of DDR4 or DDR. Yeah, DDR4 most likely. >> and whatever you have some storage.

58:14 Everybody has like 500 gigs or more of storage. So, so realistically we could actually get pretty good performance on pretty good models doing this. So, I I built this like this is all inspired by local maxing. >> but I rent GPUs from vast.ai, runpod, prime intellect, modal. So, I I'll just like drop 50 bucks in each and then have them rent like the cheapest GPUs like the RTX 2000 or whatever they have. they'll rent them and then they'll just try to get models running and optimized on each one and then they'll create a docker container which is like an executable script that loads the same model with the same configuration across the same thing.

58:57 >> So it's basically yeah a recipe it's it's guaranteed to perform at a certain like level as long as you aren't like clock limiting the clock speeds or whatever. so that I think is actually we have like 300 recipes, 60 GPUs, and I think that's kind of where where we have we we want to think about models as like game cartridges. >> It's like a CD you plug in. You don't need to think about what want is this.

59:22 It needs to work >> at a certain expected level. >> Yes. >> No tinkering. >> Yeah. >> And I think that's the best way to do it because it's guaranteed. You know, we've run it on this hardware already. And there's local.ai. Does that try to solve that or >> so? Local.AI, we were benchmarking and getting information like data around different quants, different models like like how close are they to the frontier? What can they produce? Like we were just harvesting a bunch of this data.

59:50 >> like running benchmarks, right? Evals, benchmarks, speed sweeps, and we're Yeah, I I can't say more about >> Okay, it's in stealth. All right. Exciting things to come though, I'm sure. >> Yes, this month. >> Excellent. Oh, very soon. Looking forward to it, >> dude. Opus is breakthrough after breakthrough after breakthrough every day. Like every day I expect Okay, there's there's certainly no much more we can squeeze out of this thing. >> It's and it keeps getting better every day.

60:18 >> Speaking of breakthroughs, I recently saw that people found how to unlock like is it VRAM or speed in the 3090s? Can you >> I actually have another 30 minutes, give or take. >> Yeah, this should only take like 5 minutes. all right. So, yeah, speaking of breakthroughs, I saw that recently there like I can't remember who it was, maybe you can speak to it. They found like that the hardware of like the 3090s and some of these other chips are actually limited somehow and you could unlock more. Is it speed or Yeah. Can you speak to that?

60:50 >> So, with the 3090 it was flops so it makes prefill faster. I think it was like 2x the prefill. >> 2x the prefill. So on GeForce cards, there was like some firmware. I don't know if it's firmware or software, but there was some setting that was like limiting >> how much yeah, how much the cards could tap into. And so these this applies to I think all the 30 series chips, the >> RTX, GeForce, whatever 30 series chips.

61:25 But this isn't the first time something like this happens. There was a GPU called the CH 100X or CH1X1 some something like this. CHX100. >> Mhm. >> this was like an A 100 type chip or A60 something like that. it's like ampear generation pretty sure or maybe one before that. but they were selling it as an an 8 to 10 gigabyte chip, >> but actually it has 64 GB >> that is artificially locked.

62:00 >> Wow. >> >> so this is something that like Nvidia is doing. >> This is something that is done for all hardware companies. >> All hardware companies, >> I believe. I'm pretty sure. I mean, they're like, if they're doing it, >> just extrapolate, you know, it's >> the reason they do this from what I've read. >> From what I've read, I'm ignorant. >> I mean, you can't know for sure. They're not going to Well, maybe they will, but they haven't.

62:28 >> Yeah. Well, you can kind of like >> go backwards. So, there's >> stability, >> like software stability. >> Yeah. >> Right. >> Right. CPUs like can get overclocked, but >> Yeah. ship at a certain clock speed. >> Yeah. So a lot of this hardware is older. So part of it could be stability. >> Part of it could be product differentiation. So if you are selling a data center GPU >> that has >> 80 gigabytes like an the A100 64 80 gigabytes >> and it has whatever 135 whatever flops and you could just get four RTX3090s for 20% of the price. Like obviously it's more of a pain to manage them but even those ones had NVLink.

63:18 >> Mhm. M >> so you know you know about NVL link. >> Yeah. Yeah. Like helps you chain multiple GPUs, right? >> Mhm. >> So usually GPUs are plugged into PCIe slots. Yeah. >> And those could read at for PCIe 5. I think it's 64 I don't know if it's gigabytes or gigabits per second. >> I think it's gigabytes, but it can read at 64. So the interconnect between the GPU slows down the actual task. So if you split a model on two GPUs >> 64 >> is that >> I'm just trying to think like sorry to bring you back but the speed like a Thunderbolt 5 isn't that like over 100.

64:00 I'm like >> I think it's 10. >> It's 10. Okay. Okay. >> I think >> Okay. So it's like six times Thunderbolt 5. >> Yeah. It's pretty good interconnect. it's like No, it's actually 128. So it's 128 and and I think DDR4 is 64. the raw PCIe. >> Yeah, the raw PCIe. So the interconnect between them. So if you split a model across two GPUs, like you could put half of the weights here, half of them here like or you could do tensor parallelism where you split the model like so the math is happening equally on both or pipeline parallelism where you put half here and half here and it goes like step by step >> but there's always an interconnect step.

64:35 Yeah. >> So if you are so like at the most basic you have the the board the motherboard which is either you know gen 4 or gen 5 so 64 or 128 like there's a limit how how fast they can connect. >> So if you have to go to the CPU which a lot of software just defaults to using the CPU to like schedule things right. So if if one GPU has to go to the CPU before it goes to the other GPU, which is how most most people have it set up like at home probably not many people run multiple GPUs, but you're adding extra latency. So you can have the GPUs do P2P so they can talk directly to each other and they can get more speed.

65:18 How much like I yeah they can just get optimal speed that way. and VLink like five or 10x what whatever generation it is >> it makes it so fast that it's like you have one big GPU with double the memory >> bandwidth >> so like there's no loss there's no time of >> so the 3090 had an >> the RTX Pro 6000 doesn't >> Oh >> the 5090 doesn't right the 3090 did but you know these these later generations don't >> cuz that would be a problem >> why >> because you can get by the data center.

65:53 >> Mhm. >> Yeah. >> Yeah. No, that makes that's what I was thinking, too. Like they're doing it so they can upsell. >> Here's here's another reason they might do it. So, you have all this machinery that actually like puts together like all these laser machines, all these like presses, and I don't know how much they automate, but there's definitely a lot of machinery that goes into like setting up the chip. So they they're they're probably not going to have an exact like replica of each chip. So what they want to do is have like common standards like common like we put in a board this big with this much space with this many like RAM modules or whatever into the machine so that like so you can automate the process. So you have a single like thing that's going in and then based on how you want to differentiate your product, you have processes like later on whether you maybe like remove the envel I'm speaking nonsense here but does that make sense?

66:57 >> Yeah. Yeah. I'm actually like fascinated about it more so now but chip manufacturing is like something that I didn't really consider before. I thought it was just like >> Well, I guess chips before weren't really this sophisticated. >> they were always the most sophisticated thing. >> Yeah. Yeah. Now, people say the fab is like the most complex thing like humanity has ever made, but that's like the processor itself, not the chip. Chip. Chip is I see it like pick and place. It's like a basic CAD board, but I guess like electronics on to it.

67:38 >> I think they're at like three nanometers >> for the processor >> for the trans that's >> but that's like one processor whereas like >> I guess it is chip manufacturing. >> Yeah. You you you have like basic modules, right, that are just like you stack them in a certain configuration. So, have you ever seen those people that like upgrade 4090s or 5090s to like more RAM? >> Yeah. They're just they're just like basically making it closer to what what would be an RTX Pro 6000, right? Cuz the RTX 5090 and RTX Pro 6000 have the same stats, memory bandwidth and prefill >> like flops. I think so. I know they have the same memory bandwidth. I don't know about the flops.

68:18 >> So, it's just there's less RAM modules on this one than this one. I was talking to this chip guy and he was saying that those RAM modifications are relatively so much easier than most chip mods in general. >> I don't know. I don't know. I've seen like some stuff on Twitter and YouTube, but I don't like I would be too scared. >> Yeah. >> If you think you remember chipping like PS2s and stuff so you can play any game.

68:50 >> I did. I did do stuff. Not to the degree where I would mess with hardware. >> Yeah. >> Yeah. >> But you know, people like I knew there was a person in my city that like you pay him and then he does it for you. >> Yeah. Business. >> I I know. I've I've been to one of those places. It's awesome. >> It's so cool. Yeah. He was like a little monk. >> There could be a business to do that for people's GPUs to add more.

69:12 >> There it is. It is. Oh, it is. >> There are people that already service this. It's just like you still have to get the actual RAM modules. you still have to >> right >> have like how much would it all like cost to >> I saw something like three gigabytes of the RAM that would extend a 5090 is $100. >> Oh, that's pretty good. >> So good. >> 30 GB what? Like >> Yeah. A,000 >> to add 30 gigs.

69:43 >> Mhm. But you could just add 100 for I mean three for >> I mean you then have like the issue of cooling and >> Oh really? >> Yeah. Because the like the chip >> you know like you have to cool it. There's still like limits. I mean I'm not saying it's a huge problem like it works for sure. There are people that are very skilled at doing that kind of thing. >> >> which chipset would you say? So, I know the 4090, the 5090, >> the 40 >> 4 like >> Yeah, if you go on Alibaba, AliExpress or Alibaba, one of the two, you can buy these modified chips like those CH100, whatever GPUs, the same thing. You you have to like literally do something on the hardware level to unlock it. Like they solder it off. People say they like void their warranty.

70:36 >> of course. I mean, if you're like taking like your >> How much is a warranty normally? >> So, I don't think that matters for people in Shenzen >> like the re that rebought. >> H Yeah. They they either buy them or they get like broken ones and like re like there's many ways of like salvaging old stuff that we cuz when when we throw stuff out, it's usually in China. Like it usually ends up like if it's technology like it's going to end up in China.

71:01 >> They love buying that Yeah. >> Yeah. Because it's it's >> we should we should have more people that are like tinkering >> doing it here. >> Yeah. >> 100%. >> Oh, yeah. >> Yeah. I just wanted to get the 3090 like modification part. >> Yeah. You just give it to whatever model and say, "Crack it. Let's go." >> Yeah. Yeah. >> Have you tried? >> I don't have a 3090. I got I got a 5080.

71:27 >> Mhm. >> you can just hammer that thing and see if there's any unlocks like that. That's what I like there's going to be millions of unlocks. >> I think the I recently got it and I think it's on the warranty >> and I think that >> if you just mess with software, I don't think there's an issue. Like you could have a model read you the warranty and like tell you what's >> software. Oh, you think there's an unlock like >> that's the 3090 was a software unlock.

71:51 >> Huh. Okay. >> I'm pretty sure. I'm pretty sure. You know, >> we got GLM to look at it. What do you use for reverse engineering and cyers? >> Deepseek and GLM. >> Deepseek and GLM. >> 41 flash or >> the like latest 401 flash. >> 41 1017. >> Yeah. Yeah. So good. So good. It's such a good model. Even the one before it was so good. >> And you run it locally. >> I do. It's awesome. It's such a great model.

72:16 >> That's how you know you're getting the the exact model. And then what what hardware? >> The 6000s. Four RTX Pro 6000s. >> Four 6000s. >> I have like parts as well. Full well you know the full precision they publish in four bits. >> Uhhuh. >> So a lot of people just like publish and serve four bits. No. >> Well if that's the best you can get. >> No like because of the hardware Blackwell has like 4bit tensors. So like it has like special cores for four bit operations which makes it much much faster.

72:52 >> Have you ever is it such thing as like unquantizing? >> there is >> more precision. >> There is what you can do the >> fine tuning. >> I don't know cuz you're >> you know low riders. Are these low riders? >> This gold one coming up is my favorite. >> Oh yeah, I see it. Feels like a low rider festival in the mission. That's so cool.