📆 ThursdAI - Nov 7 - Video version, full o1 was given and taken away, Anthropic price hike-u, hal...

0:00 all right folks it looks like it's working slowly it's working few folks are joining us yam is the last one that we're going to join they're going to wait for and then we're just going to go and go and have fun so I think that the agenda for today let's run to the agenda for today so for foxo listening I'm going to run through the regular sections of our show which is open source llms we have not one but we're

0:22 going to talk about the week of small open source LMS lm2 was released by huging face is new and they call best open 1 billion parameter language model and it looks pretty dope on on the metric so we're going to chat about the small LM one also I believe Nvidia respond as well I don't have it in my notes but we're going to go and find this out as well I think that let me small yeah these are the ones that I

0:46 wanted to talk about meta released mobile llm which 120 million 125 million parameters 350 million parameters 600 and 1B so those are like tiny ones that run on your device probably super fast as well and hugen face with the small LM series of models 135 million that's million with an M that's not billion we're used to we're used to talk about billions of parameters 135 million that's I don't know your MacBook M1 can probably turn out niston can probably

1:13 calculate how many tokens super quick that's that's going to go super fast and AMD also released AO 1 billion parameter Ando we know abouto all right another thing for open source llms we should talk about 10 cents I'm going to butcher this 10 cents Yan one something like this large which is a whopping 389 billion so you guys see the difference the scale the massive scale difference here we're talking about 125 million parameters to 380 billion parameters Moe

1:47 with 52 billion parameters active it's a huge language model with 206 contact window 250,000 tokens in the condex window it comparable to Lama 71 Lama 70b 3.1 and very closely comparable also to 400 45 billion parameter model for Lama it's a 10 cents open source and largest Transformer based Moe so far so that's very impressive if you guys remember llama 405b is a dense model

2:17 this is anoe and I think that's pretty much it open source wasn't like popping this week too much but then we're going to go into big companies of LMS and there has been some announcements I think the biggest it's not necessarily the N ounc we talk about everything right we talk about a bunch of stuff so openi buys and opens chat.com and I I yeah chat.com is now redirect to chat gp.com they bought it from dares

2:42 it's very interesting there's a whole there's a whole theory about like how much this actually costs I think it's around $15 million we're going to chat about this as well and I put this here I put this open the eye drops 01 full and we're going to wait for this and hit the breaking news button also all one full right now if you guys remember we have 01 mini 01 preview on the platforms and we know that 01 full exists we know this

3:07 because Sam told us in Dev day and also because last week we're going to cover this when we get there xai has now released their API we talked about this last week but now they offer 25 bucks a month for free in grock API credit till the end of the year so you can play with it and there's a whole thing to talk about Peter levels and the number of clicks it gets to get the API key and

3:28 I'm salty folks you want to stay here for this I salty about this thing also anthropic releases hiu 3.5 I think probably this should be like the biggest kind of news up here this is super dope if I'm able to like Wrangle with my notes thingy okay high CO 3.5 is super cool the price there is very interesting so we're going to chat about the pricing structure of this thing and I'm getting PS that I'm missing herds Dev so we're

3:53 going to chat about her Dev as well whatever that is yam and niston I would love to hear from you guys about Ed eted is a company that announced Sonu the first Transformer Asic and for folks who have no idea what I just said Asic is a custom based Hardware device that spits out the Transformers well etched into the device itself to the chip on the chip and then they are talking about 500,000 tokens per second output from

4:20 this lolm 500,000 tokens per second it's insane insane gpus do 100 200 the top ones do 2,000 the the lpus and the npus and whatever cerebrus has and whatever this is 500,000 we're going to chat about what does an Asic mean in the Transformers World a small thing about perplexity being valued 9 billion parameters ha and meta and scale have announced defense llama in this week's Buzz this week's Buzz a reminder this week's Buzz is a category where I chat

4:51 about everything related to weights and biases about the stuff that we have the courses that we have the product announcement that we have so we're going to do a recap of the hakatan that we had here with the it this last weekend it was dope Dex was it dope oh it was super dope awesome Dex was here and a bunch of other folks joined us as well I really enjoyed and had a lot of fun Chef made an appearance

5:12 that that we built for the hackaton previously and Chef was here roasting Lucas the CEO of w and bis got roasted and that was fun pretty spicy there yeah Chef is foul mouth as well and he uses Gemini and you don't even have to break jailbreak Gemini for that spiciness so that's that's pretty cool so Chef is actually here somewhere in in the in in the bag taking a break but yeah hack was super dope we're going

5:34 to chat about some maybe some projects the second AI Hardware device puppet that I have is fester for the Halloween toy I think I have a problem I think I need to go see a person about the AI Hardware devices Fester the Halloween toy finally had its debut if you guys remember last week I jumped on the oh I should probably stop sharing the screen I jumped on the Twitter space and the YouTube and the reason for this was

5:58 because I was wearing a costume and I wanted to show you Fester he was right here with me after that Fester actually went and saw kitties and actually did what he was supposed to do identify Customs I really want to chat with you about this I even I think I have some videos to show now that the format is multimodal I can show some pictures the folks on Twitter space I have a link for you as well I have a full write up of

6:18 how to actually do this if you want to build let's say Santa whatever for you you could you I'll send you to the write up as well and I definitely want to chat about this and let me go back to my to my thingy Super Quick Share all right you guys should see my notes and oh yeah and I want to invite those of you who are coming out to new new IPS is coming out very soon in a month and we're going

6:43 to have a party at the news it's in Vancouver Canada and we have a party there at wnb I will have another thing to announce with w and soon but not yet Ino and audio we have a tiny thing called fish agent it's super cool it's a continuation of quen 2.5 3B the Train on 200 billion audio and text tokens and it's an end to endend speech to speech model which is like an omni from open

7:07 the eye I unfortunately haven't able to run the demo but I trust VB from huging face when he says it's super cool so it's super cool we're going to mention this and maybe even try it out on Space because we may have time today and oh a and diffusion flux just killed with this ridiculous thing so flux 1.1 Pro is now HD it outputs 4X the resolution that previously did and the images of this is just absolutely stunning they are

7:34 absolutely stunning so flux is absolutely killing the game recently they have been overshadowed a little bit by by red panda just a tiny bit not a big not a much flux Pro is still the absolute king of AI art diffusion right now and the new Forex resolution is just absolutely mind-blowing so we're definitely going to chat about this maybe we have some folks in the audience I see it's a fre with us maybe we'll join who are more of experts in that

7:59 Fields I think it's most of what we have to chat about for this week let's see if I missed a few things folks I know definitely I got PS about a few things that we've missed folks on stage let's start with the folks here on on the on streamyard anything that I missed big wol from anything huge there you see I sent you a few the Moi whatever it's called the mji model it was two weeks

8:24 ago when it needed four h100 graphic cards and now it already work on just the 3090 so we are down to the recent most recent news was 12 gab vram is all you need to run this state of the art model so it's no Zora local yet but it is a big step ahead I think so I tried it and it's a great model you can get it as with a one click with Pinocchio so you can play with it on your own system

8:49 if you have a bit enough RAM and yeah that is the video model and Allama added Vision support better Vision support for the Llama 3.2 models that are fully supported now so you can run the 11b and the 90b with that as well all right there was that Herz Dev thing that you mentioned earlier yeah it's basically like another Oshi but it's newer and it seems like it might be better unfortunately I haven't seen many people attempt to test it yet

9:16 I don't exactly have much info on exactly benchmarks or anything like that but yeah I think it's wor yeah we just pull up all right folks I think it's it's time for open source AI Let's Get It Started niston you're usually the small LM guy if you want to start off let's get it started all right yeah niston small LMS I'm going to turn off my my sharing here there's a few of them I would love to hear from you if

9:39 anything of them at all are exciting for you as well there's ones from Let's see we have we have meta releasing mobile llm and very tiny 125 million parameter is there anything to do with this n like are there really useful is there is that like an autocorrect level that you can just like autocorrect stuff what' you think yeah just quick assessment just use quen coder 1.5b nothing comes close out of all of these so I tried the the amdo yeah it it

10:09 was just wasn't good I tried the 1.2 small LM was a bit better and conversational but it also seemed very trained where it was very sensitive to any form of quantization the the only useful thing is the one from Gina which is a 0.5b that auto corrects for what she asked I think we website data into XML and it actually does it way better than gb4 like much more accurately and it's just been trying to grab website

10:38 sources dumps and turn them into XML formatted and it's very good at that job and I think that's like the only one at sub 1B that actually has a practical use which you could use today for your business or for customers anything else does not does not come close yeah the small LM 1.2b was pretty good again if people don't know my most popular model was Biggie Small LM like 1. 54b and that was based on the small

11:11 LM version one one 130m I showed to be like the fastest small tiny one that you can run at two tokens per second on one no that was an evolutionary merge evolutionary mer and and it was it was trained on a whole new Optimizer so it was trained on the gro optimizer that that was a lot there that that went on into just getting that thing to talk but yeah the new ones are better however the only thing at this range like

11:41 1.5b and under that actually has very good practical use is Gus model the 0.5b that does the formatting and a lot of coding assistants or online coding assistants use the quen 1.5b coder that actually has very good inline use and I tested it on a four-year-old phone and it pulls in over 10 tokens per second so it's very practical to use and AMD is pretty not a useful model yeah it worked

12:13 but at this point yeah sorry I get impatient with these reviews is this of of any use right now and yeah now the thing about small LM is that you have access to all the data so you can match the distribution of the pre-training data with whatever else you're trying to do and you can do continued pre-training if you're studying this stuff or if you're working on on a Model that is that is a big deal because if you want

12:37 to compare as to how you continue pre-training now that you have your right data distribution you can try and and fit in your new data to the past one and get like a much better continued pre training so it does have you very useful uses in research and maybe some company F tuning things so it is worth spending resources on the small LM one but yeah so that's my I think I think I give a pretty thorough assessment of the

13:05 whole thing so far so just metrics on a few of the metrics that we saw from small LM gets 42% versus llama 3.21 billion parameter at 36% and this is a slightly bigger model so that kind of makes sense as well trivia car 36% they trained on 11 trillion tokens of curated data sets so this is like very tiny model trained for a lot of data and yeah that's pretty much it yeah I I quickly want to say that the big accomplishment

13:33 here is the data set like that is a massive contribution it's not the model itself I don't I hardly think there's going to be like over a thousand times more resources or like maybe more maybe 10,000 times more resources and time and and computes spent doing stuff with the data set release than the model that's just a proof of concept the data that's just likely delivered am mounted so that's a huge that's insane contribution on on their part like if this was just a

14:03 data set release it would have been pretty good the the model sorry honestly is just not that that good it repeats itself needs a lot of work shows some interesting things where it it's very sensitive even to 8bit quantization so just leave that in F-16 whereas other ones you could you could quantize it more so yeah but that's a massive contribution in how much data they they shipped awesome all right moving on from very tiny models to very big ones we're

14:29 getting a new a new huge model from 10cent which is called Yan 89 billion parameters it's an active I'm actually going to present and we're going to go and take a look at this together what do you guys think because I yeah's let's go take a look at this together I think should I pop a new window for this I think that's going to be a good idea and I will present meanwhile anybody already had a chance to play with hang large I

14:53 would but it's not available in Europe again I have to remove it from the map here yeah their license is worldwide just not in the whole of Europe that's pry concerning actually go ahead I did check it out and it's really interesting there's a lot of kind of architectural tweaks and changes that they made that gives me similar Vibes to deep seek it makes me think maybe they P some deep SE people or something but they did some

15:17 things like cross layer attention they had some custom learning rate stuff that were like expert specific learning rates for training and like these are all like new things that aren't really standard in training process and overall it seems I remember you were talking about the comparisons to 70b but this actually seems something more like I would say a 405b competitor and even in a lot of areas especially like math it does better than 405b and a big thing

15:43 here too is the training tokens is only about 8.5 trillion tokens and 1.5 trillion of those are synthetic data and if you do the math overall in terms of the active parameters as well as the total tokens it was trained for we're talking about something that was if I remember my math something like more than eight times less training compute than llama 405b wow so this is pretty good in terms of just training and inference efficiency perspective getting

16:12 pretty close to those results are even better than 45b while using way less training compute and much faster to inference as well since it's only 50 billion active parameters thanks man yeah running through the technical report super quick and it looks like you've covered most of the interesting things here I used it feelings about it wait yeah feelings and I want to hear from wol from isn't that wasn't it dropped in open source can't you just download no yeah you

16:39 could of course you could it's not licens of for company use it a nogo just like meta did not release Vision models in Europe the farma 3.2 versions those were not licensed here as well so that seems to be a new trend prob putting pressure on the EU so hopefully they will reconsider their AI regulation I use this for code and I just it just wasn't that good I just stick with deep Seeker so I thought at the end of the

17:06 day it ends up being about 50 billion active parameters but you still need over 80 something gigs of RAM so even if you have Nvidia G card eight of them might not be enough to actually run it it's not very practical to run and even if you're going to run a sep inference you still have 50 billion active parameters to deal with so it's going to be slow there is that and on top of that it just was wasn't responding that great

17:31 like half the time I would ask it to respond in a certain language and stuff and it would I'll put everything in Chinese and the other models inep speak can ask to do stuff in English and they do as they're told they do the job and it yeah I don't know it might I haven't tested that thoroughly but to me this is I'm not gonna not gonna really bother much there's deep seek there's quen if you need to run stuff in open source I

17:57 would use it for benar or generating some entropy data with different distribution data if you need a map or other stuff so it still has uses for data generation but for dayto day I I I didn't really I didn't really like it didn't change the landscape too much folks hopefully folks can hear me well folks are saying that audio could be a bit louder so I can project but also I added some loudness to the audio all

18:21 right I think that's mostly it on the open source some folks said HS something and I think that that's mostly it ohj do you do you remember the last thing that I can go and look up and we can take a look together yeah if you just look up her if that's what you're thinking of yeah her de yeah her that Dev I Herz Dash I think Herz okay let's do let's look at X together this one oh yeah that that

18:45 one's probably good yeah open source first of It kind base models for full duplex conversation AIO oh that's dope man oh my God can you guys hear this by way I cannot you cannot I cannot so it is a full duplex conversational audio and what we've heard by the way is two like going from one left and right channels first open source base model for conversational audio generation oh looks super cool checkpoints and and yeah you guys are going to see a little

19:13 bit behind of how I view things when I need to prepare for Thursday I the first thing I do I go into this like longest blog and I was like no I don't want to read all this so I use ARC and I sumarize in 10 bullet points this is the first thing I do and this is a beautiful thing that now I have a open source audio basee model with 8.5 billion parameters is codec LM and Herz voice

19:36 activated vae is a variational auto encoder variational Auto encoder thank you and let's see what's super interesting there we're reading this as we go along folks unfortunately this isn't as polished as the prepared news but somebody just threw it in the comments and hopefully unless some of the hosts here already have some way of anything you want to say about this feel free to jump in but base model so it's not specifically F or something I see a

20:02 p 2.0 license on hugging face so there's okay so now that that puts it on a whole different story I don't know 8.5v does this mean is this based on llama or quen it's probably based on Quin I think it's neither I think they trained it they mentioned two trillion training tokens and I think something like six million or something on the order of some millions of hours of audio that they train it on as well so yeah and they

20:27 also actually released a version with just trained on millions of hours of audio and not trained on any text at all I would like to ask one of our co-hosts to play the audio cuz I think if you guys share there will not be an issue and you can actually play it anybody wants to take this m to play as a sample super quick with one and two channels n you want to take the the you want to try

20:48 one I have the entire OS sandbox so don't make me troubl shoot that right now please anybody would like easy setup D just can click and play so you mean just go to the website and yeah share your like window when this website is on and then just hit play and then it will come through to the to the folks wol you want to you want to try I can try if I have so many windows wait a

21:08 second I need a new window for that all right folks just a second we're trying new things we're trying I can do a audio pause no I'm just kidding but basically what I because the show is now multimodal as everything else this is a new voice kind of thing end to end voice thing and I played one sample but you guys couldn't hear this and that was on purpose because I also bring you never mind so I'm on the page now audio is on

21:35 so you should see it any moment now yeah I think we're we see hit play okay so I will hit play about this stuff every time in so many different terms and if you're in the young I'll take a look at some of the different ones but with all of that said this is another new episode of fall so what you heard folks is generated audio from hers thing do you mind hitting the two channel one yeah 10

22:01 9 8 7 6 5 4 three dope I like it nice stereo yeah nice stereo a little bit what's the interactive one probably people talking let's see welcome to the standard intelligence podcast we have a very special guest with us today his name is Bob how's it going Bob yeah I'm doing pretty well I'm very excited to be with you all on this nerdy podcast yeah we'll talk about a lot of stuff like AI what are your thoughts on AI Wait f

22:32 pause pause this for a second that's really good I don't know folks who listening just on the Twitter space can tell the difference when we stop talking and that starts folks I don't know maybe leave us comments and tell us but that sounded dope as hell I wonder if that was a person talking to the AI or two AI talking like in a podcast I can't tell it's so good wow that was good that was really good that was really good that

22:55 was really and it's a licensed okay okay all right okay we have to move to video next we cannot stay only in audio land audio land is scre let's cover the let's cover the fish one too yeah okay so this was this and then what else is on this page super quick so that folks what else is very interesting here during live inference model needs to run 84 passes blah blah it's super boring latency oh latency super cool latency is 62

23:21 milliseconds average time between given utterance and the end of one token that's incredible running our servers close to the end user we have a achieve the real world average latency 120 milliseconds Jesus so folks I always say this but I'll repeat this all the time 12 12 is instantaneous everything below 200 milliseconds is feels instantaneous to humans because this is how this is the research Google did once when you click on the button and it's less than

23:49 200 millisecond latency to reaction you feel that it happened instantly 120 milliseconds for a audio to get back to you is it's what happens when you see it a human it reacts it's really less and it sounds super dope so I'm very excited to be able to play with this Apachi to license how big is the model can we download this it's less than 10 billion parameters it's I think they said about 8.5 yeah yeah I would wait until someone

24:14 converts it to safe tensors just because I'm paranoid about running direct PT files but it's all in there yeah the download checkpoints link on their website leads you to a another link oh it leads you to a txt f on their server index.txt that lists out five links to PT files what the hell is this I'm kind of like nerdy like okay that's good that's good that that's a good thing actually that yeah I'm into this who's

24:43 the guy in charge with hey dude let's put up a downloadable txt file outlining six PT links in the txt file that's that's a very converted way to download this all right folks so Herz ddev is a endtoend audio full dlex model that less than 10 billion parameters that was able to generate incredible podcast sounding audio with multiple channels and thanks to wol we were able to actually hear this and it sounded dope it sounds like notebook Alm to an extent like an

25:12 open source notebook LM all right LJ you want to cover the other one that we had also because fish oh sorry fish oh no no fish speech spe yeah they trained one using yeah yeah fish is also also a similar thing I didn't hear fish though I didn't hear samples of fish but also an end to endend speech to speech model looks like we got two so yeah but I haven't heard it at all being at all

25:42 good it's okay but there's a reason I'm a great fan of it is because you can actually swap out the 3B model that's inside it with your own fine tun and that makes it a lot more utilitarian for whatever doing because you could just have your your model trained for whatever you can even swap out like the the speech part and yeah and that's also that's CC by NC so it's actually not not they use but they used quen 3B and the

26:12 3B quen has one of the highest GP QA Diamond scores of any model under 8B so it even like matches the newr ab which is like really good the 3B coin the only one of the small ones that they up license there's a reason why it's really good so this one uses that that specific model and it makes it very practical for on device use because that models run great on device and in my own setup just

26:42 using whisper with which you can get whisper with a little bit of training and some engineering you can get whisper to 200 Ms total latency and then you have the latency of of llm and you end up with 300 something so you can do that with just engineering and not using an endtoend train model but in this case they paired it together for you so it's yeah in some ways this is better I just wish it the license just makes it hard

27:08 to use because I I would bet you anything like nine times out of 10 the the 3 bcoin Model is going to be a lot better than whatever any other organization trained at seven or 8B so being able to have that in there and being able to to find tune that is is a lot more useful to me but the no license so that makes it not useful anyway yeah the the speech is not is great but I

27:35 would rather from hers what do you think because I I was wasn't able not even close not even close no it's it's not bad yeah her is Aion too it's not fish is not bad this is actually not bad at all the but it's I still I would much rather use the fish model because as a 3B assistant that's like very good has insanely high scores you would be able it would be a much better rag agent to

28:01 as a as an as a chatbot that you would use like during work tell me oh how do I do a Singleton in like elixir or something if you have random questions like that you want a smart model to begin with but you also want to be really fast and that's what I would use I would trust the qu models to do that job but it's not licensed for like actually making a product for it that

28:25 sucks anyway yeah do use I'm a fan of what they've set up with that fish because I I was ready to just rig that on my own and I'm just glad someone else did it oh that's dope so I think folks we've covered like two segments in one we've covered open source Ai llms and Audio models herds Dev thanks for the folks who suggested to cover herds Dev I haven't saw it yesterday and some folks as always folks the community is

28:50 incredible not only to mention that we have folks on stage here who joined from the community now co-hosting this but also the fact that we have a bunch of folks I I see maybe I'll shout out yeah I should definitely shout out folks in incredible folks in the community that are always supporting I see Lincoln is incredible and always brings me the links that in covers you definitely should give him a follow I'll he's in the community probably going to be a

29:13 moderator in the community I see Bowski dropping some conversations Nicole Roman I see Thomas is joining us as well a bunch of folks herzfeld Labs Unity Eagle janed is here A bunch of folks Ray Fernando in the chats I really lesan in Twitter spaces dazzling Tony Ryan it's incredible to see theic like across all of them I would love to hear from you guys what format do you prefer because the audio format we just saw we're

29:41 basically cooked like AI can take over that this the smiles that we're having I don't think AI is quite there yet we're going to talk about the video model but it feels like we need to upgrade at least to me it feels like we need to upgrade so I think we're moving yam I saw you struggling to join us for multiple things you want to give a you want to try to unmute mute here and and

29:58 tell us what you think about the Audio model or any comments that you had throughout the stuff that we talked about maybe the Transformer stuff you have too many Cuda Colonels running at the same time oh really interesting folks yeah looks like looks at least the folks that I can hear him I can hear him on the phone yeah looks like we're good or at least some of us are good some of us some of the time not all of us all of

30:18 the time all right folks I think it's time to move on to big companies and the discussion around their stuff I think it's 9:22 I think that we're actually going to move there in around an hour I will also say that as we have a bunch of folks looks like we had 800 views on the live YouTube stuff oh sorry on the live X stuff I don't know how many folks are watching on YouTube some of them maybe

30:41 prefer on YouTube we also have a bunch of folks in the Twitter space as well maybe it's time to do like a little bit of a reset hey folks you are on Thursday I for the first for the second time but for the first time coming to you with multiple folks as well on coming to you in video and audio thing we're still struggling through some technical as we always hopefully at some point be this clean but hey it works at least some of

31:04 the time yeah okay the reset you guys on my name is Alex volov and the I evangelist with weights and biases for those of you who are seeing this you can see the title like right here this is super cool for those of you are hearing this on the Twitter space this is the view of our community Twitter space which is awesome we have now a community on X with 1600 strong which you can post links over there if you don't want to

31:27 spam your main and you want to just go deep into a subject the community is a great place for that feel free to do if you want to moderate. community definitely reach out to me and I will consider it if you're a great member of Thursday ey and you you haven't been on the spaces and you're not like super public you know don't want to talk but you want to help I would love help in moderation of the community if you want

31:48 to produce just send content there you can cross post to your main enter the community by the way that's a new feature would appreciate that and you'll get some exposure as well we have been talking about open source AI for the past 45ish minutes the rest of the conversation is going to be about some other stuff we're waiting for open the eye to drop some news they're usually doing this at around 9:30ish I think that is the usual thing they may or may

32:12 not drop some1 stuff we're going to cover this in a second and I I definitely want to cover some weights and bias of stuff in the next segment and also remind that I have a guest here Dexter horthy and we're going to chat about how agents interact with humans yeah what's up y'all super stoked to be here yeah what should we do next I think that we'll do a little break for the weights and bias stuff and and I think

32:37 yeah I think let's do the weights and bu of stuff super quick and then I'll show some examples live on the you stuff so if you're listening on next and you want to jump and onto the video stuff and want to actually see what's going on this would probably be the right time for that yeah all right cool okay I should I have a transition for this let me see if I can have a transition for

32:56 this this we bugs from Thursday no this sucked I I'll prepare this for this later this this was really bad really bad folks so this box is a category where I talk about everything happened in Thursday in weights and bises this week that was interesting to me and the main thing that we had this week was obviously the hackaton together with AI the hackaton with the i tinkerers that we hosted this weekend the hackaton was called the human in Loop hackaton and

33:25 shout out to Joe from I thinker who collaborate we collaborated with Google Cloud was here brought a ton of Swag just a ridiculous amount of Swag we got out swagged in our own office folks I don't know if this happens ever to anybody else we got out swagged in our own office which led to some conversations with some with some folks we will not be out swaged again I will tell you that and and it was dope we had more than 100

33:50 I think around 120 people came in to hack and we had a great super energetic weekend if you follow some of my stuff we definitely had folks building incredible things all the meeting rooms were full people gave a bunch of comments folks were building around the topic of human in the loop which is also the topical conversation that that me and Dex will have Dex I'm going to I'm going to add you to the stage somehow

34:12 I'll find a way to add only you let's see how we'll do this oh okay I'm just going to remove the other folks and all right folks so maybe do a little introduction and then we're going to just move from there yeah no I I think the hackathon is great I love Joe he he put it in a way that I probably have been saying for a while but not quite as eloquently which is this idea that AI

34:33 agents are this New Concept and people are trying to figure out what what are they and and what do they look like and everyone's got their own definition and I think the most important thing that J honed in on and why the theme of the hackathon with humans in the loop is this idea of AI agents are almost good enough they're almost good enough to do really impressive things and go beyond just text to text or human sitting in a

34:55 chat interface they're almost ready to be unhooked and sent off into the world to just be in the background building doing things taking care of business and by finding innovative ways to bring humans into the loop we are able to shortcut that waiting for the next big model waiting for the next big thing and get real results and and build agents that actually do things in the world do things that are maybe even scary like update production databases or send

35:19 emails on my behalf it was really cool I love the community I think in building human layer the the the people that identify the most identify with the most are those people who are in the trenches just building stuff that they want for themselves whether it's assistants that work over email whether it's I don't know there were so many cool projects what was your favorite project this week so I was a judge as well and the

35:41 coolest one of the coolest things I think it was second place I was partial to Hardware stuff somebody built a it was pre-built a little bit a kind of a hardware toy that looks like a little Cactus the moves it was for kids to talk to and it talks back and and I think it was called Toy Story 6 and it basically is for clinicians to try and have kids talk about what they went through identifies them and talks back to them I

36:08 thought it was super cool just basically the The Voice based conversation were were super cool I love the idea of being able to like pause agents and go and for for additional conversation folks getting like tell me that your audios little quiet let me boost you up a little bit and yeah so one other standout hackaton project for me was somebody built a lab assistant I think in in complete augmented reality with the Vision Pro where they kind of

36:34 basically those crazy for reparing a lab and combining some components they added an augmented reality thing on top of your hand that tracks in real time whether or not you took enough of one component and enough of another component like it turns red in real time if you took more than one so that was super cool yeah PR were great judges were awesome the energy in the office was incredible we will do more folks so if you're listening to this you're like

36:59 oh that sounds cool we are like committed to having an open community space in this in office of our so we will do more if you need to tell your boss hey there's a reason for me to go to San Francisco you can learn a bunch you can meet other people this is always great I I'll be definitely talking about the hackaton that we're doing here at w Andes and I loved the fact that I got to present the the chef

37:22 puppet that you guys may be familiar with the robotic arm that we 3D printed and it got to roast a bunch of folks live and also I did a raffle for The Meta Ray bands which I'm not going to wear on stream some of you know why because this thing takes over some Bluto stuff never mind but I did a raffle and the chef actually roasted the whole crowd and said you guys are all lazy why are you here listening to this

37:44 go back to hacking and then announce kind of the winner so that was really fun so that one thing that happened to W and B this week the other thing I wanted to cover is is fester and I really want to want to show this super quick this going to be let's see I just want to run through this share screen and then window and then the second window this yes all right wb. mehow weave so I

38:08 posted this fairly extensive article finally about about how hacking a skeleton to detect kids costumes and GRE them with a custom spooky message not my best title work as well Hacker News didn't pick it up but yeah you guys remember those of you who are here last week for those of you who weren't here last week I dressed up as a m scientist and and and my lab invention was this skeleton cold Fester we're probably not

38:32 going to be able to hear this be able to see this so I just wanted to show you actual reactions from folks it detected with I want to say 65% like accuracy that was pretty good this is a shark in Eeyore he detected this I love this cute other met scientist that showed up to my door and actually took a selfie with them because I was I think he went as Einstein I'm not sure but Fester the skeleton that I

38:55 built the uses Gemini also shout out to Gemini all of this cost me exactly zero by the way from the LM perspective which is incredible Fester was able to identify kids custumes which was its whole point I posted finally my run through of everything of how I built this folks so if you're interested in doing skeleton brain surgery as well I recommend you go and check out this video it's on wb. mehallo weave I'm going to post this

39:23 on the stream as well and then you guys should be able to see this in comments yeah folks are saying that Fester is dope I really appreciate everybody who enjoys Fester give it a like on YouTube cuz you got exactly never mind so Raspberry Pi is controlling this Gemini visual for the brains I took over the LEDs I explained the process of how I learned to do this no I believe no soldering is essentially required if you

39:47 really don't want to and I yeah my kids have interacted with this and then I shared some more examples of folks in trick or treat we're past Halloween a little bit I I was debating whether or not post it before Halloween so people can see it on Halloween and get excited or after Halloween with actual examples I'm very happy that I posted after Halloween with actual examples H the reason why this is again in the way and

40:08 bias coary is because this project was called hello weave and yeah this this Fester this is how my I'm showing for the folks who are only listening I'm showing my very messy desk with all the tools that was needed for this you don't have to go as messy by the way but all of the software requirements and by the way all of the stuff that you need to buy to aeve achieve something like this are also linked here and the reason why

40:30 I'm bringing this up in the weights and biases Corner as well is because I 100% was helped by weep Fester here's an example of fester I'm showing a picture of fester standing outside of my door on Halloween Halloween night waiting for kids looking creepy as heck and then not working and the only reason I was able to know that it's not working the camera was obscured or something like this and I have an example of this here hopefully

40:55 folks who are watching the screen can see as you can see some of the streams that I had weave just showed me like a where the picture was supposed to be is just a blank picture and I noticed this I was like oh the camera doesn't work and the problem with okay so the problem with Gemini is that even if you pass a blank image and you tell it hey identify the Customs for these kids Gemini will do it Gemini will say oh no

41:18 problem man oh welcome spooky ghost whatever and no there's no one there it will just absolutely hallucinate and so it's really hard to know know if it or not you can you need to listen so we was instrumental funny we was very instrumental for me to actually debug this and you guys can see the date on this 4:47 p.m. trick or trick started at 5: some people come before but most of the trick-or triers show up at around

41:42 513 minutes before production the fact that I had weave in a hardware device disconnected I wasn't like even s back then I was just looking at the logs help me catch a bug this is a screenshot from slack of me thanking the team on actual like dday of of Halloween that's it I wanted to share this with you guys super quick and this is this is the write up wb. me/ weave I'm going to put it up on

42:06 the yeah I'm going to put it up on the banner so folks can see it like here and set it up and hopefully you guys can see 1 b. mehall weave check it out and I think we're back now to chat about everything else what do you guys think yam unfortunately is not joining us on the video we we're going to set it up separately and yam will join us next time I promise or are you here yam maybe

42:32 no I don't know yeah the nist's stand Standing joke always applies we get AGI before we solve all these zoom zoom zoomi issues folks it's time for us to chat about big company API updates and I think we're going to start with we're going to start with Haiku right Haiku is the first thing that we should talk about so who wants to pick this up let's let's do a quick summary of the changes we know we knew about hio 3.5 for a

42:59 while because when sa 3.5 V2 or sa 3.6 as some folks like to call this was announced from from anthropic they also said hey hiu 3.5 is coming no mention of oppos and they didn't say when and finally Hau 3.5 was now available the API it's available everywhere on workex it's available on obviously on Tropics API platform as well and available open router shout out open shout out open router folks

43:31 thoughts I want to hear thoughts about 3.5 hio before we go to the technical thing I don't have that that many the only thing about technicals is it's more expensive than the previous hu which is a also something I want to I want to chat about to go first I can tell you that I have been using it since Tuesday so I made it my main model actually and have been using it productively in over 110 chats which

43:54 yeah at work at home all the time so I really wanted to test it but unfortunately I noticed it rides very well but the intelligence is definitely lower than zet of course we expected that it's cheaper than that despite being much more expensive than the old Hau but it's not prompt following well enough so either it doesn't understand the prompts or it doesn't follow as well so I had a lot of problems I was using perplexity and

44:20 they had it in there as well so I switched to that and I had to regenerate with Sonet sometimes relatively often actually where I noticed okay it's not doing what I told it what has worked before so regenerate with onet and it was working again which is a bit disappointing because they say that it's better than the old Sonet before the new release that came out in October 22 but I can't say that it is better than that

44:46 I don't have the feeling because I was using Sonet 2.5 the old one all the time and it was better than what I saw now with Hau and yeah that was my test my experience in in real use and I also tried it with computer use because I thought it is cheaper so it could work with that but it doesn't support it yet unfortunately that may be for for this model yeah and it doesn't have image input as far as I know that

45:11 doesn't working either yet wait no image input yeah I think it doesn't have image input I'm not 100% sure but I think I read that it doesn't support it so I haven't tried it with the images I can check it again if it's true yeah let's take a look let's look at some of the numbers GP Diamond compared to the previous hu 3.5 gets 41.6 compared to GPD 40 at 53 so obviously not there but this is a tiny

45:37 like a super fast model as well on MML U pro it gets 65 and let's see what's interesting for us sbench verified the agentic coding it gets 4% so that is very decent I love the fact that they're adding they're adding that it's surpassing oppus Cloud fre oppus do you guys remember the oppus used to be like this insanely great models super expensive and we're like oh oppos look so good whatever this tiny model I can't say this is like super cheap model

46:04 cuz the price is I think $1 per million tokens and then $5 per output million so not like extremely cheap obviously not Gemini flash level but still fast and cheap still fast and cheap it beats oppos from eight months ago and oppos was back then we were like oh oppos is like incredible so definitely the the acceleration is palpable over there I keep thinking about oppus 3.5 and why oppus is not coming up LJ did you find a

46:31 way to raise your hand even in this yeah so with Hau it's I've noticed they say that it scores better than Cloud 3 Opus overall and benchmarks and I think that's believable but like walr said the the intelligence or reasoning seems to be lower and I think what's happening here with the benchmarks is it's significantly better in I guess you could say memorization and pattern matching and then just also worse at the kind of more around intelligence

46:58 reasoning but then the the balance out of what ends up happening in the benchmarks ends up a little bit over Cloud 3 Opus just on average and I think the types of downsides that you actually experience in real world tasks are going to be in things like very long context and multi-term conversations a lot of back and forth coding if you want to do that's probably going to be much worse and then certain things just like World

47:21 Knowledge will probably be things that it does better than Cloud theopus yeah and the new Sonet that folks starting calling 3.6 you guys you guys okay with this naming are we okay with this I I I decided I'm going to call the the new one 3.5 and the old one 3.4 but it looks like we're we're getting towards 3.6 as 3.6 as the new on it that seems to be landed okay I I remember some controversy around this that seems to be

47:48 the the the model that folks are using anything else on high let's take a look any more thoughts niston stepped out a little bit Dex have you used the new H at all no I haven't haven't had a chance to play with it yet all right you mention PR yeah so $1 per one million input tokens and then five per million output tokens I believe it's 5x the previous one or some yeah that sounds about right but it's still

48:10 cheaper than son right signicantly yes it's still cheaper than son the the the thing is let's talk about the price hike for a little bit ldj because we expect models to get cheaper this is the expectation open I set up with every model released by like 80% price decrease and a 90% price decrease in like 99% they they they they keep doing this as far as I'm remembering this is the first major lab that released a

48:36 model that's like superseding the previous model and didn't keep the price not only the same price actually increased price from the previous one specifically for Hau Hau the whole offering was it's fast and cheap and it's it's a smart and compatible for some stuff but it's fast and cheap and Hau doesn't compete with Gemini Pro and doesn't compete with 40 or definitely not 01 Hau competes with flash hu competes with mini those are the this is

49:05 the playground of Hau and I think that they're they're not price competitive with flash at all flash is insanely cheap flash is free for literally I just said this for hallo weave the project that I did and for Chef I paid zero dollar I was on the free tier and even if I wasn't on free tier we had Simon Wilson here on the on the chat as well on the PID tier you get like million tokens for free until you start paying

49:28 for flat it's Google is basically hey here free intelligence for you is like as free as air basically I got to wonder who are they're trying to get with this pricing and especially with the increase in pricing and how much that thing will work for them thoughts folks we love to hear thoughts I think maybe they just need to really get their revenue up they need to compete more with open AI kind of show that they have cash flow going

49:51 to investors and actually the head of Applied research at open AI recently commented on the situation on Twitter and have the Tweet pulled up he basically said Jimmy Apple said hope open Ai and others won't follow unusual indeed and then Boris power the head of Applied research at open AI replied it seems very important to never do certain things if you are to remain a trustworthy platform that other companies can rely on this seems like one of them it sounds at least open AI

50:17 people are firm on not wanting to do these types of things yeah yeah is there anything do you think this has anything to do with I've always looked at Claud and anthropic as like premium in some way or another is this them just trying to preserve their Vibe as the best class in whatever category they're in could be although I don't know if Hau is actually best of class in the category of fast and cheap because cheapness is part of

50:40 the category of fast and cheap all right folks so yeah definitely price hike Haiku is still decent it was always very decent there was a period of time where Haiku was like the only fast and cheap and like very smart model it's no longer the case I'm very partial to flash flash is just incredible and like pre basically free and so I love using flash myself all right let's move on let's move on can I just add something yes

51:01 because it is writing German very well which was a very bad point of the old one so even if it's a smaller model its language capabilities are great and just to confirm so I checked with the API just now and you can't upload an image which is funny because I did while I was testing it but it's perplexity switched to gbt 40 in the background I just noticed what happens because I could upload images no it just switches

51:26 the model so it's impossible to use images with it yet but it's coming they said it's coming so in the things that in the area of things that are coming Vision related as well we now have access to Via API and we already had access to gra V API so why are we announcing this again they're not paying us oh nobody pays us oh I think we have yam on video yeah what's up we made it

51:51 finally did you reinstall the Linux or something to get here I switched to this extreme yeah yeah I see you awesome mic I don't think we can hear you from the awesome mic but we can definitely see the mic you can't nothing works pretty much nothing works yeah all right welcome y folks it's so good to see folks on stage we've been hanging out for more than a year together I think this the first time we're like hanging

52:09 out on stage so this is awesome little celebration for us and I think I I like this we should continue the next thing that I wanted to chat about is yeah X Gro we're getting access to it we've gotten access to it before the experience they had was scky the reason was you would sign up and then immediately tell you hey your organization is out of credits immediately there was like a red banner on top your organization out of credits

52:32 that was like a experience immediately straight up and then let's say you wanted to pay none of us want to pay straight up we want to play with your stuff before we even get excited we all have open router keys anyway we know that Alex atala is going to go and buy your on the back end going to buy your API so the reason for me to go into your platform is just because I'm very interested I know some people will go

52:51 directly so showing me a red banner out of the blue is not so they understood this they got the feedack back and now everybody gets 25 for free on the gro until the end of this year 25 bucks a month for free credits I believe the gro is around I believe the grock is around $5 on input and $10 on output something like this let me take a look I have this in my notes and not super cheap you get

53:14 access to gr beta whatever that means it's not yeah 128,000 tokens contact lengths function calling system prompts and vision capable coming next week so this is like in the topic of what's coming next week Vision capable is coming next week and it's only until the end of 24 $5 per input $15 per output which is like GPT for all level which is but as we've said before the benefits of using Gro is it's less censored they are

53:43 purposely building this to be less censored and so wol from I would love to hear from you on that experience if you had a chance to use it and ldj maybe you you raised your artificial hand so go ahead go first yeah I think it's just maybe worth clarifying real quick we're talking about gr with a K and not grock with a Q just in case any gr Elon musk's like X open the eye competitor that is

54:04 going to they're trining the next generation of it right now on insane amount of Memphis which is maybe why they have the pricing at this point because maybe they're trying to prepare people for so so it doesn't look as big of a jump when they release their next 10x model orever supposedly December supposedly either the training run is going to end in December or even the model is going to be something around December as we said 100 do we know

54:31 details the only scale of the Memphis super cluster we know is that 100,000 h100s were going live and then they mentioned that they're going to add it's going to be a total of like around 200,000 h100s because they added H2 200s or something so it's going to be an insane massive it's one of the biggest ones they scaled it super quick those of the we know we don't know anything else we know the ga is multimodal because

54:54 vision is now possible Vision was added by the way last week to the UI as well but on the API level we just got access for free to their to their stuff anything else to add so like you I was very excited when I read the the tweet and system prompt support and everything so I immediately went and had to log in and again although I was logged in X and it immediately said not in your region again it's bad but you

55:20 know what I realized what is what may be a real problem is what happens if you get banned for using if you get banned on X you can't use that or if you do a chat with your AI and try to uncensor it or do anything and you get banned for that so the connection between Services just like when you are on Amazon you use a TV and you have the shop and you get banned for problems with the shop what

55:44 happens then so that is something where I'm not so sure if I really want to use everything from the same provider if entropic bans me okay I can still use my Google Mail what happens if Google bans me for any reason can I use my AI can I still use my m so that that is something we should also keep in mind and speaking of uncensored or more open models I think if you prompt it right Sonet is so

56:07 uncensored it does stuff I got another Warning by them by mail that my account is now flaged and say inject we had that another time the topic where they inject stuff like talk ethically and so on so I had my AI expanded system prompt to detect those injections and rejection and report them to me and that is working as well with the good models so yeah you can use every model and they they have a lot of content I must say

56:33 that is you would be surprised if you just un unchain it unleash it yeah so you could do this in other models it feels like grock's like proposition is that it's by default like this right and this is like what we talked about I didn't really think that way because when I was using it with my character and stuff it was it didn't embody it as much so it said oh I'm just an AI and stuff like that despite the system

56:57 prompt so I can't even say for those purposes I say role playing and stuff like that I can't say that it is more in Center maybe this it less Vogue maybe that if you try to make it a character and stuff CLA is better so CLA is better for role playing even uncensored thought if you uncensor it with a good prompt it's much better interesting all right so this was the xai gro updates for us and vision is coming next week as far

57:22 as we've heard let's talk about open the ey so we waited no01 fo for n yet but we know the 01 fo exists because some people got access to it so this week 01 fo was dropped by mistake or not mistake probably somebody got access to O and full and not only they got access to O and full and folks remember we don't we try very hard to not report on like speculation or conspiracy theories on

57:46 Thursday I we try to report on like what actually happened this actually happened folks like plyy the Liberator and some other folks PL already Tred to jailbreak 01 full there's screenshots of 01 full we know that it's live in open the eye for a while we heard rumors that it's waiting until election is over and I said the word and but that's why we're hoping that it's going to come come today maybe it still will 01 what

58:10 currently we have is 01 mini and 01 preview so hopefully we'll get 01 mini 01 full this is the reasoning model from openi what we know from the screenshot is that 0 is and they've announced this in De day 01 will support function calling which it doesn't support right now it will support hopefully system message which doesn't support right now it will support Vision which is very important and the screenshot that I saw was somebody uploading a screenshot of

58:34 something and saying say oan what is this andan starts thinking about this folks I'm assuming that you saw it as much as I saw it the brief eight hour period or 3our period that somebody got access to 01 full expectations from that model to complete the break ldj go ahead yeah I think a lot of people should keep in mind that the jump from 01 the jump from GPT 40 to 01 preview is almost the same as 01 preview to 01 in a

59:02 lot of use cases for competition math and competition code here for the qualifying exams for the math Olympiad for example it goes from 133% to 56 or sorry about 133% to 45% when you jump to 0 preview and then it jumps all the way to around 75% for full 01 and then for competition code it's around 62% for 01 preview jumps all the way to 89% for Owen fo so yeah I think things are going to get exciting I I don't know how to

59:33 raise hands in this so anyway let's you said something interesting let's be honest what do you feel in terms of jump from GPD 40201 what do you think if we open this subject already you have use cases where you really feel the difference yeah yeah I will say that I often find myself I can give a practical use case for folks who are listening maybe this will help I am hiring for a social media role here at w bases for

60:01 some reason I was given the full control of social media so if if w b reacts to you in a Funny Mimi way that was me on Twitter and for some reason you think it's a good idea we'll see if it turns out to be a good idea so I'm hiring for for a social media role and I'm scanning I'm I'm I'm going through a bunch of CVS and I'm taking notes in my conversations I noticed that what I do is I take the

60:22 transcription of the whole conversation that I had and I need to answer a bunch of questions and i d the whole transcription in in chbt I noticed that oan does a better way of a better summarization of what I need from this transcriptions substantially better more thoughtful more more direct more and it's like a long convers not super long 45 minutes I notice I do both and I noticed that owan just gives me better answers I can't tell you exactly why I

60:46 know that for does a good job owan just feels this feels more I wish that I had a one with the search because search is dope and I really like the ground this is what I wanted to say you can't have o1 with advanced voice mode for example you can't have it with search and I really hope that we're going to get everything everything together in the same all models all features because I don't know advanced voice is really

61:11 useful for me this really useful advanced voice mode is incredible I keep using this all the time but advanced voice mode with one would be you ask it the question and hey let me think and then you have to go for 45 you can use the normal voice and yeah you need to wait a little bit but the normal voice yeah in the in the app it works I did it today did not know this all right folks

61:31 here you are if he didn't change the model in the background I did it today but in the app it works interesting all right ldj go ahead ldj hand raising here yeah in the recent Q&A that open AI actually did on Reddit I believe they did mention they plan to bring in the other things I think they even mentioned certain other stuff that they plan to bring to the ow models and they even said or Sam said in

61:57 a recent interview about how they plan to bring images and he alluded to almost image generation it sounds like he was alluding to as well and saying that we should expect big steep improvements to images soon is what he said and he's vague about whether he meant image perception or image generation but it sounded like he was alluding to generation so image perception from the League this week we know that it's there right now1 and one me don't support

62:24 image input from the this week somebody uploaded a screenshot we saw it does and that would be great image perception or like image generation there is a very famous if you guys remember Greg Bachman posted this about 40 not 01 an image somebody writing something on the on the Whiteboard and the person writing you guys remember this it was like an output of an image and everybody's what the and then nothing happened since then and for folks are listening not

62:48 understanding what we're talking about this is not Del the model that gets prompt in text mode from the model and gets like a API call this is is not what happens on grock on X where you ask it hey generate the image of trump kissing CA sorry I had to I said this is most of what people use grock for in image generation and then you get Gro calling in text form the API for flux and flux

63:10 generating image it's not that this is the model itself knowing how to generate images as well as output text right so full multimodality on the output as well I tried to coin the term MMO multimodal on inputs and outputs and so we are getting multimodal output models on multimodality of text and audio now already the advanced voice mode for example that's an omni model that outputs on go this is the O of gp4 O yes also has o it's not the same it it's not

63:42 a nomy model I don't know I'm not sure I heard Sam Alman talk about the o1 in 01 and the O there I think it's it stands for multiple things but specifically the joke there is that this is the joke and Sam Alman confirmed it that it's a alien of extraordinary ability o1 is the type of Visa that many people in the US get the joke is that alien of extraordinary ability o1 is alien also o1 is like

64:09 zero1 because like people say oh when they mean actually zero and and 01 they basically reset it's like a new kind of model and this is going to be 01 and 02 they confirmed there going to be like the GPD series will continue like separately so they did confirm this but I heard Sam Alman confirm this in front of sigil when sigil W is a engineer in San Francisco here who does the 0 Visa support community so if you have an 0

64:32 Visa he knows who you are and you know who he is he's giv out these hoodies if you saw most of the people in South Silicon Valley there were these hoodies say ellien of extraordinary ability I saw when Sam and him chatted about those hoodies and Sam Alman confirmed that the joke that went about those oppos before they released the o1 publicly on the Twitter that said I'm an alien of extraordinary ability and 0an is the

64:53 type of Visa that that these people get yes y I saw your face I everyone could stop the the confusing names of Sonet 3.5 Sonet 3.5 again and GPT for and O it would be just name it in a different way just so it won't be confusing but yeah yeah I loved when we had Claude Sonet Claude Haiku and Claude oppos that was incredible that was like a like it was incredible move and then they should

65:21 kept kept this going let's go and then we'll move on yeah sure I remember in the recent Q&A that they did too there was also somebody asked an open AI employee that was like I think it was Mark Chen or somebody that was a vice president about what does the o stand for and their official answer was open AI like literally o stands for open Ai and I guess there's also the extra meaning in puns of ow and visas and

65:45 stuff but I think what yam was initially I think what you're about to say is about how 01 might have omn modal abilities and I think that's still true because if you guys remember some people were getting emails on the initial 01 preview release about stuff like please don't try and jailbreak GPT 40 with reasoning and they were originally referring to it as GPT 40 with reasoning which kind of seems to entail that it's this large fine tune over GPT 40 which

66:14 would then mean that because it's the same architecture naturally they would be able to have those things like image generation and other 40 abilities yep thank you ldj for that and then I think just insert something yeah go ahead W and then because what came to my mind now you have more or that o1 is better than 40 I totally agree with that but what would be interesting of you or the others if you have compared it to the

66:39 new Sonet for example because in the Benchmark they said it reaches even the 01 levels and does any of you have tried that compared it I sometimes use the API of 01 mini just for the steps and then I use that in LM Arena to just ask son and some other model to follow those steps and then do do the rest of the problem so their stepping the way that they handle that is is still topnotch no

67:05 nobody has has done it anywhere as good I know a lot of people are trying their new models out coming with more Chain of Thought stuff but not nothing comes even close so it still outputs the best steps in terms of like how talented is it as a developer to solve this very particular issue it still does feel like just asking gbt for all sometimes like I've seen the particular solution repeated even though I asked o1 mini and and 01

67:29 and Sona did obviously quite quite a bit better I've seen some interesting things that sometimes Sonet on the desktop takes some time to answer and I don't know if that's their devops versus using it on the API on AWS it's often is quite a bit worse in API to the point where I just end up going on again on LMC Arena and just using it there if it's something that I don't care about it being public so yeah that's that's my

67:54 take on it I don't think the model self is necessarily smarter it's their operations and their order of execution and their stepping is is just on another level and also just for folks a reminder for folks 01 with reasoning sorry so for o with reasoning is 01 the reasoning behind the scenes we don't get to see but like folks who work at op see the actual reasoning and it's quite awesome but what we do get to see is another

68:21 model summarizing that reasoning and so even that is helpful newon like even grabbing that and sending this to Claude and saying hey this is like the steps of reasoning give me the output even that is helpful on its own folks let's move on because we also have a I want to chat with Dex about some stuff but this is the last thing and yam this is I saw you go hard about this you want to introduce

68:41 what's going on here right basically in a simple sentence there is a tradeoff but for folks who can't see the screen tell them what they're seeing okay you are watching the world's first Transformer Asic pretty much in simple words as simple as possible there is a trade-off when working on Hardware you print stuff on Silicon it's physical location there is a trade-off so you either have a more flexible chip that can do all sorts of instructions and all

69:08 sorts of things or you focus it on a different on a specific use case for example here you focus it on a specific architecture so it's less programmable but you get a much much more powerful chip that can do that thing so there is a tradeoff what I'm saying and some people agree I think the Transformers aren't going anywhere in my opinion people think that they will might go one day I don't think so I think they're

69:35 here to stay and what we see here this product bets on this exactly and pretty much burn Transformers physically on a chip in Silicon and you can see the performance gains at the beginning of of this landing page you can see the big performance gains that are expected from this I just want to say is an statement wait hold up hold this is an understatement oh yeah FKS we're talking about use h100s or eight times B1 b200

70:03 the most insane machine that Jensen stood on stage Nvidia being the biggest company in the world probably because some of because of that specific thing they're getting four tokens per second on llama 70b maybe 500 this is 500,000 this is like what I don't know three ERS of magnitud more like two three ORS of magnitude more I'm not good on my wombs but this is a lot of wombs a lot this is insane jumping speed because

70:28 the literally Transformers all of the thing yeah my question to you is that is the transformer in the architecture but you can put different models or is the model on the architecture look I don't know the the exact specifics of what exactly how is it implemented but just from what I've read it's pretty much designed for Transformer based models you probably can load what no you can you can load whatever model that you want and as long as it fits the AR

70:55 ecture that it supports so I assume for example that things like mamba or state space models probably are not supported but it's not I don't know I don't know I don't want say anything that is not not accurate but I suppose this is not meant for that and it's okay this is exactly what you're getting for what you're getting and what you're getting is the speed up that you all see I just want to point out two things first if you want

71:22 to see the same thing that already happened couple of years ago in a different field you can look at crypto mining Bitcoin mining specifically that started everyone started with gpus and specifically Bitcoin mining went all in on A6 and just you cannot mine Bit specifically I'm delivering the same specifically Bitcoin because every other coin it doesn't work with this because exactly when you work with an ASC you are burning something thck into the Silicon so if you if you are working if

71:54 you're mining B you can't mine ether so what I'm saying is that it already happened in in a specific field you Niche down your hardware and you get a tremendous amount of speed up second thing I want to say there is a by the way there is a large ecosystem here of AI Hardware I know at least 12 startups working on AI Hardware over here locally so I talked to some people about this already it's not magic Hardware is not

72:18 magic usually when you see a huge boost in something usually there is a cost and and I'm not saying I'm not saying that anything here is misleading it might be the best chip in the world I'm just saying that it's not magic for example you cannot train with it there might be latency there might be latency for loading the model it might be it might consume a lot of a lot of energy but there are trade-offs for anything

72:43 because it's not magic at the End Hardware is hard you print something and and it's just physically there you can just reprint whatever you want and but it's great to see in my opinion it's great to see it's time that the field matured that much that we start to see specific Hardware with neural network architectures burned in Silicon it's amazing to see in my opinion and gpus were made for games and we're still like they're no longer I wouldn't say that I

73:13 wouldn't say that I wouldn't say that about popularized by games yeah yeah what I think is important to say that gpus and gpus and so on are probably going to Ste dominate the training while you are going to see inference chips such as this competing with one another because just because but I'm not sure you're going to see as6 for training because this is pretty much T of and what is going on this is pretty much a

73:40 TPU already if you think about it yeah all right LJ and nist would love to hear your thoughts super quick and then we're gonna move on and chat with Dex about human layer and then we're going to close out the stream because we're almost at two hours yeah so I think a big thing we didn't mention yet is the a demo which is the Minecraft AI demo that a lot of people may have seen online that was I think a collaboration between

74:01 dayart and et. which is this company making the A6 and it's not running on these chips yet I don't think they have them made in production or deployed yet but they say the Demo's currently running at 3p with a very short context length on h100s but once they do have the etched AI chips out and running these Minecraft AI simulations they think they can get it to 4K resolution in real time with multi-e context length and so that will be really interesting

74:29 and actually something that people might just do for fun rather than a cool kind of gimmick right all right are you guys ready for me to wrap this up yeah Nan give us I think ET is a okay so I'm biased a bit because I've reviewed for other pretty well-known investors a lot of Chip startups and the vast majority is a lot of BS so I am biased against in that matter and I think Ed is a complete

74:55 scam to be honest it's not a real Chip it does not exist at all they just paid this other company Oasis to make a joint blog post it doesn't even run on their Hardware they don't even have a demo board running on their Hardware it's all just hypothetical just another FEC shiny website product made to be sold to other investors and this is not going to go anywhere and so far I don't think I've been wrong on these predictions I think

75:25 this whole thing is complete BS they have a small chance now that they have 120 million in funding to maybe find the right people to hire it's just that knowing actual fpga devs and the how small the community of electrical engineers who understand the stuff is I'd say good luck but yeah this is not not real and the if you look at the pictures of the die of the 800 and you look at the rendering that Ed has been

75:50 using this time you can see a lot of similarities there it basically just looks like someone took that in blender and and Sh shap things around just look at the andom tech h100 die it's basically that and just added some freaking stars and stuff in the middle so that's gonna be my public on the show folks this is exactly right thank you I I'm not affiliated with anybody this is this is what I think and if I'm wrong

76:14 you can always just show me the working ship yeah and the folks from me listening to this and you want to prove this wrong show up with the chip would love to see could I push back on on this a little bit sure one last one folks we need to move on we actually do need to move on come on one last one okay I guess real quick would be I think it's just meant to be artist interpretation on that and

76:37 I think maybe they should make it that more clear that it's not the real chip on the website but they do seem to have some cracked people that are actually in the industry and experienced in chip development at places like Nvidia and other companies on their team and the CEO himself wrote some of the back end for Apache TVM and some cool things like that so I think people should look at the team behind the companies look more

76:57 at the details like that and yeah I think it's a hard just to like nist has carful language but I will say coming into this not knowing anything I saw the website I was like oh as6 is here maybe I can buy this soon now realizing that this is like a 3D printed designed virtual thing and maybe there's not a chip this feels to me that's not right to to at least tell it's fine if you

77:21 want to invest or raise money but you need to clarify that that the St you are is the level yeah yeah there are I got excited by the 500k and I love the wet blanket that n thr this specifically for me because 500k absolutely if you also specify numbers where the numbers come from numbers are yeah we like numbers are are thing if these are numbers it's very hard to to make this and I think we have

77:47 to respect all the work that people like Salas and and tense torrent are doing and they're actually shipping them and it's very hard to make the A6 and you're just putting up I think it doesn't acknowledge the massive amount of work that that people who who are shipping actual Hardware put in it and also they it doesn't there are many challenges with actually doing it in Hardware that even cerebrus is facing with getting longer contacts to work and

78:12 the moment that you leave the chip and you go into memory you're still faced with the memory wall problem which is also big famous video on asianometry you should really watch that the AI memory wall problem from a year something ago so understand what actually this entails and yeah please don't just make claims because I'm just gonna hold them out pretty and in an very unbiased way as well which I appreciate you for all right folks moving on and this was most

78:42 of the coverage for this week the last thing that we haven't covered and we don't have time for necessarily but Mochi the video model now is down to 12 GB and soon to be run on Hardware consumer devices and now I think we're moving moving on to chat with Dex about some of the stuff that I heard and agent stuff as well and then we're going to we're going to close out after that but definitely stick with usz I'll do a

79:03 recap of everything we've chatted about as well looks like my computer is like a little bit slow this time I'm going to send folks to the backstage so you guys can go and feel free to go to the restroom whatever if you want all right folks for those of you who Dexter this is your first time on the show Absolutely welcome feel free to introduce yourself to the folks who never heard from you and then don't know

79:23 who you are and what you do yeah totally thank you and this has been super fun super thank you so much for having me on the show and yeah I'll just say for for human layer we're building an API and SDK that helps AI agents contact humans for help feedback approvals whatever they need and it's really built for what I think is the future of AI agents AI applications I actually think the word Asian is super overloaded and at this

79:44 point we spent the last three months spending more time talking about what it means than like how to build them and how to make them do useful things but what I believe is the future is what we call these outer loop agents chat youi is very common for AI applications but what I'd like to see and what I see more and more people building are these agents that run out in the world they're completely disconnected from a human

80:04 they're out there managing their own time maybe even managing their own token spend performing a variety of tasks and when you take the human out of the direct supervision that you get in a chat UI then you need new sets of tools and we were building these sorts of agents at my previous startup and we realized oh everyone building AI agents is eventually going to want to deploy them to the cloud in the back ground and

80:25 at that point they're going to need ways to get back in touch with humans we were building this squl Warehouse janitor bot that would drop a table if it hadn't been used in 90 days like I'm not 100% confident in this technology we're a two person startup so I don't have time to spend 3 months like getting all the evals in rhf right T that it's never going to do something bad let's just rig it up with slack when it wants to do

80:45 something scary it'll ping us we can review it we can respond and it's getting towards what I think is going to become a bigger and bigger Paradigm which is like uiis or appless I've been calling it where because you have language models behind everything you can build a really great really powerful application that has no web app has no UI and just communicates with people where they are whether it's over email over slack over SMS over what that's the

81:09 dream of human layer we're building some agents getting in the trenches to learn more about what is on The Cutting Edge but at the end of the day we want to be the platform that enables people to build agents that do bigger things more detached and that feel more like this true virtual teammate virtual coworker kind of experience yeah so I want to so thanks for the introduction human lir your role in human lirer I am the

81:33 founder CEO CTO we build all the things oh awesome awesome how long have you been doing human there for so we started exploring this I basically became obsessed with this around like early August and spent two three Works weeks building the MVP came out to San Francisco launched it with the AI tinkerers Community another shout out by the way shout out to Joe yeah Joe's done an amazing job of of cultivating a community where every single person you

81:56 talk to is super technical like actively building stuff there's no you go hang out a Discord there's a lot of like hype people and you can see the lineage from the crypto to the meme stocks to the nfts to like now there's a lot of that flowing at that Ai and Joe's events are super high signal and so we launched that after 3 weeks and got a ton of PLL ton of great conversations like we found

82:18 our people and so that was when we decided to it full time and let's help everybody build so here's how Dex ended up on the show and folks there's a maybe a lesson for me in there but definitely a lesson for folks who come up to me we chatted for a brief period like I you said human whatever Dex is higher than me which is not very common for me because I'm 63 and I was like all

82:39 right let me chat with this tell dude and then at some point they were like hey Dex is like hey maybe you can maybe I can come on your show and I was like see now you've lost your chance usually when when folks ask me to come on the show that's usually when when I'm saying nope was the role right if you ask to come then you can't yeah usually and then during the hakatan the hakatan

82:59 theme was human in the loop so many folks built things where a human was required this was like the theme of the hakatan that you build some something that runs and then the human gets involved in some sort of way which is perfect for what you're building and then like many people use human layer and I think I saw a demo from you either talking about this yeah I think you give a little demo yeah and I looked at this

83:23 and I was like oh that's super cool and then we chatted and he actually showed me like a cool demo with slack as well and I was like oh that's super cool and I was like I I would love to chat about this the podcast so this is the way this is the way show me something cool and I would love to share it with you folks and what what I loved about this specifically is the idea of look I want

83:41 to run stuff and I have a few projects in mind that I would definitely want to have this intelligence of mind that runs somewhere that does something to trigger and say hey Alex wherever I'm out in the world and you shout it out hey which platforms would you guys want and I immediately sell telegram because telegram is the one that I automate a lot and the API is great super easy you can probably Bing it out in day send me

84:04 a telegram message and say hey Alex what do what do I would love this from an EA that if I have an EA I should probably get one I would love this from from somebody that helps me on different things and I would love this from my AI that that I would run and this seemed to me the best way to achieve this now here's my question to you I could technically write this up feels like

84:24 that I could rig this up with some function calling I don't know exactly how but what do you say to folks who are like Technical and they can they think they can write this up because I have the answer but I would love to hear what you answer yeah it's something I thought about a lot in the first couple months of building this and cool took a couple weeks we can rig it up with this lack

84:44 API we can catch web hooks we can fire that back down to the agent I will tell you after the last month or so of building things it gets really complex from a AI agent like context window management and figuring out which messages go to which agent and all that kind of stuff I just spent the new thing I've been obsessed with in the last like week is agents that operate over email so what I've wanted more than anything

85:06 else for a long time is something where I can just forward an email to a specific email address and that goes into an agent's context window I got this email from our compliance provider and they said hey you need to file your California annual compliance report there going penalty and I have a guy who does it I have an operations contractor he loves doing this kind of stuff I don't know why he's awesome I forward him the emails take cont but he's

85:28 part-time and I'm sometimes wondering like what he's working on or when things are what I really want to do is like when I get one of those things is I go into linear which we use for all of our projects and I create a ticket and then I ping him on the ticket and then I also forward him the email and it's just like a lot of it's enough work it's not hard but it's enough work to take me out of

85:43 flow and I'm just trying to grind through my email inbox it's what I'd love to have and what I'm like working on probably this afternoon is I want to forward an email to some special address it goes into an agent that agent calls some tools and does some stuff and then when it's ready to do the scary thing like create a ticket or send someone else an email on my behalf it gets back with me on email and says hey here's

86:02 what I'm going to do and I can just like explicitly approve or reject that action and if it has questions it emails me back as well and so I can hold a conversation with the agent asynchronously it's running out in the world and this one of the core problems that I've been really excited about late recently yeah I think that answers your question you got there and then you moved into an email thing and I appreciate this because I have multiple

86:23 emails that love to just send and for additional contexts but yeah back to why is why wouldn't you just build this yourself once you want to do slack and email and maybe Multiplex across multiple people and say all of these two people and any of these four people have to approve this before it can go through it becomes a do I really want to like spend all my time building like approval flows or do I want to build my product

86:45 that's my answer weights and biases weave which is observability framework which you should use absolutely people have been doing this for themselves right either people are logging and saving these logs in log files or people are dumping this in a sequence file whatever some people are building like a rudimentary UI on top of this and people are saying like like some people like why do I need this and the feature set that we've gotten to there's no way for

87:08 a company to build this and the product it just there's no way because the amount of complexity the fact that we are like always online with the SLA like all these things infrastructure as a service basically is is on top of it there's a best UI on top of it so I I absolutely have an answer to this question I just wanted you to to verbalize this but then you pivoted into something I would I would love to talk

87:30 about open interpreter which you probably have you heard about open yes I've heard it a bunch of times remind me which one that is open interpreter is the the software that runs on your Mac and executes commands on your Mac like computer use but on your Mac and does things on your Mac also from tiners in Seattle and computer use that that from an Tropic all these things they have something that runs on your computer and it's scary as hell and the

87:55 way the entropic dealt with this they said hey we're releasing this agent or not even agent the model that knows how to do this and then we don't give you a tool to do this we give you a Docker container only for developers whatever and it only runs in a Docker and don't put any they're like super scared about like what this could do and they put this in Docker and then also it asks you some stuff but mostly it doesn't ask you

88:17 anything and it expects you to be there while it does its thing open interpreter also multiple ch GPT stuff they expect you to look at the thing and like it scrolled yeah you click a button to approve the in the and and you wait there while it does its thing and I hate it I'm done waiting like I don't want waiting and you're talking about async yeah async means I'm free to do whatever I want until and you're switching the

88:43 formula basically like the agent waits for me and I'm not waiting on the agent and that's what I found novel and I think this is the reason I want to talk to you would love to hear more about this because are there any kind of Dev kind of also does that a little bit Deon goes and does its own thing and you're like all right I'm going to go and come back and does like full stack of

89:00 software I don't have access to Deon by the way Stephen please give me access but this is at least the approach of these like newer tools unlike cursor they like they go they do a bunch of things and then you come back I definitely want to think of a world where this happens more than I have to wait for them have you seen some of the stuff that you built what actually is practical in the real world for you at

89:20 this point that like you are able to take off your plate because you have now stuff that are asynchronous and you can go and enjoy your life yeah great great question so like some things like lead research I think that like building Robo sdrs for very specific use cases to go search very specific places where your users are we're at the point find leads go scrape a couple web pages to get their email or contact info go all the

89:43 way to the point of drafting the email and that's when I get a slack message that is hey here's the person here's what I'm thinking and here's what I'm going to send them and I've gotten to the point where after working with it for a couple weeks and I at the beginning I was rejecting almost half of the messages like this is wrong or the tone is weird or it just still even when the bot doesn't hallucinate even when it

90:02 calls the function right sometimes it just sounds bad and I'm not comfortable having to go out with and so we R those exactly and I'm the type of person that I don't have the patience to like build a giant eval set and test and test and test I want to YOLO things out keep an eye on them make guarantee nothing bad is going to happen and then as it's operating in the real world on production data I am constantly every

90:23 couple days going in updating the prompts and tuning it and now it's to the point where I accept 90% of messages 90% yeah what changed I just updated the prompt a bunch oh you so you like you keep sending yeses and then as you know what happens and so like you know the examples I'm seeing every single thing it gets wrong and eventually we'd like to build features to automate that throw the bot in a Channel with your four

90:44 salespeople and they're reviewing all of that and suddenly you created like very highly motivated data labelers because if they don't give feedback in the moment to the bot then is going to go out with their name on it that's not yeah the additional so first of all yes absolutely they have human labeling built in there you I'm thinking of like where's the wheel of improvement that's automatic that you don't have to go and change the pump follow up on one thing

91:08 is that the idea came to me that when I discussed my project with you that I want to build you're talking specifically about approval or denial of features what I wanted to chat about you about is additional context yeah approval denial is like some sort of context in my head and the project that I wanted to I'll just tell you folks if somebody wants to build this we have a gab repo for weave and that Gab repo

91:32 folks are working like crazy they release a bunch of stuff and they cut a release and that release notes is vast developers can't be bothered to put like very nice deformed message of what they built and sometimes it's connected to J ticket it's another URL sometimes whatever and I want to automate some of that to be able to deliver to developers who are reading those news or users who use weave like at nice formatted change log that makes sense for them even the

91:57 screenshot I think you suggested like browser base whatever even the screenshot of what's new what's new on a practical level I want to automate this with AI I should I think it should be possible and in my head ingesting some of this and then running through some of the links whatever already automatable to a sense but then I want the human in Loop to review this the human in Loop or maybe multiple humans in Loop like maybe

92:16 the product manager Etc because it's obvious that the sloppy machine can hallucinate and then just say something that's wrong and we're like oh no that we didn't release the sorry folks so you can't automate this fully you can go all the way to prepare a draft and you want to approve or accept this but then on the way I want the additional context from the folks I maybe want to find the developer that's tacked on on

92:35 the gab Bish Etc how are you thinking about additional context on the way to approval is that a feature that built in or is that like just a use case of human layer yeah and it's a great question yeah these are the two main form factors that we' built in and actually I'll add also in the approved reject workflow when you reject something you're required to give feedback because the only way you're going to get a better

92:54 answer is if you tell the llm like why it's wrong because then it can try again and usually incorporate that feedback but yeah the other thing we have is so that's transparent to the LM the LM thinks it's sending an email it might not even know that human approval is happening until the rejection comes in but the other one is if if an NM outputs a tool called output structured output that says hey I need help here I want it

93:16 I've been told in the prompt to get feedback or I've tried calling a bunch of tools and none of them are working and I'm just stuck or I'm in a browser session I need a two a code and I need to pull it a human to come in and get me unstuck and then I can proceed oh that's one just oh cool okay yeah but it's the LM is deciding to call a human and so you can human layer lets you expose a

93:35 variety of tool you can give one tool to contact one person or you hey here's five tools here's how you can contact the head of opson slack here's how you can email the CEO whatever roles that you want to give the lon access to you can code that in awesome and additional context as well all right nextx I think tell me where human layer is going from now on and how can folks find you and how can they use this and and we'll

93:57 close there yeah I'm Dex fory on Twitter you can find me on LinkedIn we are online atum layer. and we are again I love the AI Community I'm working with at some point we're building a venture back business so we have to find a way to make money but my favorite thing is getting in the weeds with Indie hackers who have really interesting use cases who are building really groundbreaking things with AI and on on the where we're

94:19 going part I was I was going to try not to say the a word but I I'll put in this context you know Sam keeps talking about AGI is going to be looks something like a senior engineer that you can just give some task and it'll go off and figure this out and come back to you with an answer and I think what we have today is more like summer interns and if you ask me I would never I would say hey look go

94:39 research all this stuff draft a bunch of emails and then come sit with me and I'm going to review them with you and then we can go send them out and you'll learn over time from those reviewing and coaching sessions and I think that is a big step towards something that feels a lot more like a virtual teammate like a virtual coworker and so the future is we want to build we want to be at the core

95:00 of the next generation of really awesome AI experiences and middleware middleware layer stuff that gets in the way and that specifically is oh that's exactly what I need for some of my projects as well a agents mean a lot of things as well and for a lot of them getting to hey stop and ask the human is something that people will maybe don't even know they need and when they do need they don't even know that when people try it

95:25 for the first time they're like oh now I get it and then they text me 2 days later it was crazy I had an agent running on my laptop and I went out to get food and I it slacked me about something that I needed to pay attention to and I was able to pull right back in and I was like that's the magic and we want to build more of that yeah so we just launched email this week we've had

95:41 slack for a while that one's kind of the tightest and most thought through sounds like we need telegram for to make your stuff work and then more looking at Discord as soon as someone offers me enough money we'll build Ms team support I don't know when that's going to happen and then we also have a private Alpha for RCS SMS WhatsApp these more consumer basing things yeah messages supports RCA or whatever format that they not support

96:03 with Android RCS is yeah it's going to beage soon I think it's like end of the year all the carriers are planning to have RCS going amazing all right so Dex thank you for coming to Thursday I folks this was Dex from human lir go follow him wherever he is if this fits your agent use case give human layer a shout out as a reminder if I ever do sponsor content all on on Thursday you will know

96:24 I just met Dex and I thought it was cool I thought that many of you will enjoy this on on the stage and maybe for your projects let me bring some of the folks back up wol and niston as we as we kind of recap my my Mac seems to be doesn't want me to do this I'll bring you folks back up ldj looks like it's dropped all right folks we've been at this for two hours and something welcome back up and

96:48 ldj will join us as well let's do a little recap super quick and then we're going to let you go for the the rest of Thursday we did not get 01 do we oh breaking news Well at least news news research announced you can just chat to their model now on you can go to.new research.com so that's hmes news.com and I think that's their 70b I'm trying it now it's pretty good yeah it's just their 70b yeah so

97:15 they they launched their own UI and I think you can wait is that for yeah I think so I don't know you want look at the it looks like it's a nexts app all right F the Cod CH it doesn't look like it's Forge okay I'm waiting for Forge Forge is the one that I'm that I want but let's see if this is it let Lo in it's like free of charge I can just yeah all right it

97:43 works I just dumped in a prompt I literally just tried it yeah I Tri it now yeah cool wait okay so that's somebody wants to show it I wish I could yeah all right where's the reasoning stuff is this it I keep waiting for Forge I don't know if this is the forge stuff system prom deductive no I don't see reasoning here okay so I I'll be waiting for Forge from news research where you can actually see reasoning steps and that's

98:06 something that they've announced so shout out to we going to going to collect such a great data with this this is such a great mechanism this a great way all right folks have a happy Thursday we'll see you next week byebye week bye thanks y