Section Insights
Pros and Cons of Open Sourcing AI
What are the pros and cons of open sourcing AI?
Open sourcing AI can democratize access and innovation, but it also raises concerns about safety and control, especially with powerful models like AGI. The speaker expresses a general favor for open sourcing but acknowledges the complexities involved.
- Open sourcing can prevent monopolization of AI technologies.
- There are significant compute requirements for developing AGI.
- The transition from open source to closed source in AI companies can lead to ethical concerns.
The Evolution of OpenAI
How did OpenAI evolve from its original mission?
OpenAI was initially founded as a nonprofit with a focus on open-source AI, but it has since transitioned to a closed-source, profit-driven model. This shift is viewed negatively by the speaker, who emphasizes the importance of maintaining an open-source ethos.
- OpenAI's founding was influenced by discussions on AI safety.
- The original intent of OpenAI was to promote open-source AI development.
- The shift to a closed-source model raises ethical questions about AI governance.
AI in Social Media Recommendations
How does AI handle social media content recommendations?
AI systems must efficiently filter and recommend content from vast amounts of data, which involves complex algorithms to determine relevance and user preferences. The current systems struggle with recommending content from accounts that users do not follow.
- AI must process hundreds of millions of posts to provide relevant recommendations.
- Current recommendation systems are limited in their ability to suggest content outside of a user's immediate network.
- The speed and efficiency of AI algorithms are crucial for real-time recommendations.
Complexity of Content Analysis
What challenges does AI face in analyzing diverse content types?
AI must navigate the complexity of different content formats, such as short tweets versus long videos, and extract meaningful insights from them. The speaker suggests that AI could analyze video content to provide personalized recommendations based on user preferences.
- Different content types require different analytical approaches.
- AI could enhance recommendations by analyzing the context around video content.
- Personalized recommendations could be improved by AI understanding user preferences deeply.
The Role of Regret in AI and Human Experience
What is the significance of regret in human experience and AI?
The discussion touches on the concept of regret and its implications for personal growth. The speaker humorously suggests that AI, like Grok, could have a perspective on regret, but ultimately, AI lacks personal experiences and emotions.
- Regret can be a meaningful aspect of human experience that contributes to personal growth.
- AI does not possess emotions or personal experiences like humans do.
- The humorous exchange highlights the limitations of AI in understanding human feelings.
Transcript
0:02 - Yes. - What are the pros and cons of open sourcing AI to you as another way to combat, you know, a company running away with AGI? - In order to run like really deep intelligence, you need a lot of compute. So it's not like, you know, you can just fire up a PC in your basement and be running AGI, at least not yet.
0:33 You know, Grok was trained on 8,000, A100s running at peak efficiency, and Grok's gonna get a lot better by the way. We'll be more than doubling our compute every couple months for the next several months. - There's a nice writeup Aha went from Grok-0 to Grok-1, - Like Grok? (laughing) - Yeah. Right Grok just bragging, making shit up about itself. (laughing) - Does Grok, Grok, Grok. - Yeah. (Elon laughing) It's like a weird AI dating site where it exaggerates about itself.
1:08 No, there's a writeup of, you know, like where it stands now, the history of its development and where it stands on some benchmarks compared to the state of the art GPT 35. And so, I mean, there's, you know, there's Llama, you can open source, once it's trained, you can open source a model. And for fine tuning all that kind of stuff. Like what to is the pros and cons of that, of open sourcing based models?
1:40 - I think the similarity to open sourcing, I think perhaps with a slight time delay, you know, I don't know, six months even. I think I'm generally in favor of open sourcing, like bias with open sourcing. I mean, it is a concern to me that, you know, Open AI, you know, I was, you know, I think, I guess arguably, the prime, you know, prime mover behind Open AI in the sense that it was created because of discussions that I had with Larry Page back when he and I were friends and you know, I stayed at his house, and I talked to him about AI safety.
2:21 And Larry did not care about AI safety, or at least at the time he didn't, you know, and at one point he called me a speciesist for being pro-human. And I'm like, well, what team are you on, Larry, on Team Robot? And I'm like, okay, so at the time, you know, Google had acquired Deep Mind. They had probably two thirds of all AI resource, you know, probably two thirds of all the AI researchers in the world that basically infinite money and compute.
2:54 And the guy in charge, you know, Larry Page did not care about safety. And even yelled at me and quote me a speciesist, pro human. - So I don't know if you know this, a lot humans, they can change their mind. And maybe you and Larry Page can still can be friends once more. - I'd like to be friends with Larry again. He got, really the breaking of the friendship was over Open AI. And specifically I think the key moment was recruiting Ilya Sutskever, so.
3:34 - I love Ilya. He's so brilliant. - Ilya's a good human, smart, good heart. And that was a tough recruiting battle. It was mostly Demis on one side and me on the other, both trying to recruit Ilya. And Ilya went back and forth, you know, just gonna stay at Google. Then he was gonna leave, then he was gonna stay, then he was gonna leave. And finally he did agree to join Open AI. That was one of the toughest recruiting battles we've ever had.
4:05 And, but that was really the linchpin for Open AI being successful. And I was, you know, also instrumental in recruiting a number of other people. And I provided all of the funding in the beginning, over $40 million. And the name, (laughing) the Open in Open AI is supposed to mean open source. And it was created as a nonprofit open source. And now it is a closed source for maximum profit, which I think is not good karma.
4:38 - But like we talked about with war and leaders talking, I do hope that there's only a few folks working on this at the highest level. I do hope you reinvigorate friendships here. - Like I said, I'd like to be friends again with Larry. I haven't seen him in ages. And we were friends for a very long time. I met Larry Page before he got funding for Google, or actually I guess before he got venture funding, I think it, he got the first like a 100K from I think Bechtel Zy or someone.
5:07 - It's wild to think about all that happened. And you've guys known each other that whole time, just 20 years. - Since maybe 98 or something. - Yeah. It's crazy. Crazy how much has happened since then. - Yeah, 25 years. A lot has happened, it's insane. - But you're seeing the tension there. Like maybe delayed open source. - Delayed, yeah. Like what is the source that is open, you know what I mean? Like there's basically it's a giant CSV file.
5:37 (laughing) - Yeah, yeah. - With a bunch of numbers. - Yep. - What do you do with that giant file of numbers? You know, how do you run, like the amount of actual, the lines of code is very small and most of the work, the software work is in the curation of the data. So it's like trying to figure out what data is separating good data from bad data.
6:08 Like, like you can't just crawl the internet 'cause there's a lot of junk out there. A huge percentage of websites have more noise than signal. You know, they're, or 'cause they're just used for search engine optimization. They're literally just scam websites, so... - How do you by the way, sorry to interrupt. Get the signal. Separate the signal and noise on X? That's such a fascinating source of data. You know, no offense to people posting on X, but sometimes there's a little bit of noise.
6:39 So what... - Yeah, I think the signal noise could be greatly improved. I mean really, all of the posts on the X platform should be AI recommended, meaning like we should populate a vector space around any given post, compare that to the vector space around any user and match the two. Right now there is a little bit of AI used for the recommended posts, but it's mostly heuristics. And if there's a reply, where the reply to a post could be much better than the original post, it will, according to the current rules of the system, get almost no attention compared to a primary post.
7:20 - Oh, a lot of that. I got the sense, so you, a lot of the X algorithm has been open source and been written up about, and it seems there to be some machine learning. It's disparate, but there's some machine. - There's a little bit. But it needs to be entirely that. Like, at least in the, like, if you explicitly follow someone, that's one thing. But if you, in terms of what is recommended from people that you don't follow, that should all be AI.
7:45 - I mean it's a fascinating problem. - Yeah. - So there's several aspects of it that's fascinating. First, as the writeup goes, it first picks 1500 tweets from a pool of hundreds of millions. First of all, that's fascinating, 'cause you have hundreds of millions of posts every single day and it has to pick 1500 from which it then does obviously people you follow, but then there's also like some kind of clustering it has to do to figure out what kind of human are you, what kind of new clusters might be relevant to you, people like you.
8:15 This kind of, this kind of problem is just fascinating. 'Cause it has to then rank those 1500 with some filtering and then recommend you just a handful. And to me what's really fascinating is how fast that has to do that. So currently that entire pipeline to go from several hundred million to a handful is takes 220 seconds of CPU time, single CPU time. - Yeah. - And then it has to do that in like a second.
8:44 So it has to be like super distributed in fascinating ways. Like there's just a lot of tweets, there's a lot. - It's a lot of stuff on the system. And I think, but I think it right now it's not currently good at recommending things that from accounts you don't follow. Yeah. Or where there's more than one degree of separation. So, you know, it's pretty good if there's at least like some commonality between someone you follow like something or repost it or comment on it or something like that.
9:19 But if there's no, let's say somebody posted something really interesting, but you have no followers in common, you would not see it. - Interesting. And then as you said, reply, like replies might not surface either. - Replies basically never get seen. 'cause they're never, they're, currently, I'm not saying it's correct, I'm saying it's incorrect. Replies have, you know, couple of magnitude less importance than primary posts. - Do you think this can be more and more converted into end to end neural net?
9:52 - Yeah, yeah, that's what it should be. - So you can... - For the recommendations should be purely a vector correlation. Like there's a series of vectors, you know, basically parameters, vectors, whatever you wanna call 'em. But sort of things that the system knows that you like, and like maybe there's like several hundred sort of vectors associated with each user account and then any post in the system, whether it's video, audio, short post, long post.
10:29 The reason I, by the way I wanna move away from tweet is that, you know, people are posting like two, three hour videos on the site. That's not a tweet, like it's very, like Tweeeet for two hours, come on. Tweet made sense when it was like 140 characters of text. 'Cause it's like a bunch of like little birds tweeting. But when you've got long form content, it's no longer a tweet. - Yeah. - So a movie is not a tweet.
10:55 And like, you know, apple for example, posted like the entire episode of the Silo, the entire thing on our platform. And by the way, it was their number one social media thing ever in engagement, of anything on any platform ever. So it was a great idea. And by the way, I done, I just learned about it afterwards. I was like, hey, wow, they posted an entire hour long episode of. So, no, that's not a tweet that, you know, it's a video.
11:21 - But from a neural net perspective, it becomes really complex whether it's a single. So like everything's data, so single sentence, a clever sort of joke, dad joke, is in the same pool as a three hour video. - Yeah. I mean right now it's a hodgepodge for that reason. It's, but you know, like if, let's say in the case of Apple posting like an entire episode of this series, pretty good series by the way, this Silo, I watched it.
11:51 So there's gonna be a lot of discussion around it. So that you've got a lot of context, people commenting they like it, they don't like it, or they like this or that, you know? And you can then populate the vector space based on the context of all the comments around it. So even though it's a video, there's a lot of information around it that allows you to populate the vector space of that hour long video. And then you can obviously get more sophisticated by having the AI actually watch the movie.
12:19 - Yeah. Right. - And tell you if you're gonna like the movie, - Convert the movie into like, into a language essentially. - Yeah, analyze this movie and just like your movie critic or TV series, and then recommend based on after AI watches the movie. Just like a friend can tell you. If a friend knows you well, a friend can recommend a movie and with high probably that you'll like it.
12:50 - But this is like a friend that's analyzing whatever. - It's like AI... - Hundreds of millions. - Yeah. Actually, frankly AI will be better than, will know you better than your friends know you, and most of your friends anyway. - Yeah, and as part of this, it should also feed you advertisements in a way that's like, I mean I like advertisements that are like well done, right? - Yeah, yeah. - The whole point is 'cause it funds things, like an advertisement that you actually want to see is a big success.
13:18 - Absolutely. You want ads that are, advertising that is if for a product or service that you actually need when you need it, it's content. And then even if it's not something that you need when you need it, if it's at least aesthetically pleasing and entertaining, you know, it could be like a Coca-Cola ad like, you know, they actually run a lot of great ads on the X system and McDonald's does too. And, you know, it's that they can do, you can do something that's like, well this is just a cool thing.
13:54 And, you know, so you're not, basically the question is do you regret seeing it or not? And if you don't regret seeing it, it's a win. - How do you, so there's a bunch of signals that are incorporated, hearts and reposts and maybe number of seconds you linger on a post or something like this? - Yeah. Attention is a big factor. - Attention. - So that's why it's like, it is actually better to do things that are long form on the system because it basically is ting up how many user seconds, you know, users were interested in this thing for how many seconds.
14:32 So if it's a really short thing, well they will be less. Like if it's a link leading out of the system, which we're not opposed to at all, it just is gonna have fewer user seconds, then if that article was posted on the X platform. - How hard is it to go from maximizing minutes on platform to maximizing un-regretted minutes? - Well, I mean, our aspiration is certainly un-regretted minutes. - It's a tricky, it's a big, it's a tough...
14:57 - It is. - Like, it's a fascinating leap. 'Cause regret is a fascinating concept. - Exactly. - You know, happiness. - I have a joke with a friend of mine. Like, you need this drug called regretamine. You just take one pill or one dose of regretamine, and all your regrets are gone.(laughing) Wait, do you want to have regrets though? So then I'm like, maybe you wanna have regrets, you don't regret anything, was anything meaningful? - Yeah, there's a French song like that.
15:25 - You "Non, Je Ne Regrette Rien"? - Yeah. - I regret nothing. - Yeah. - (laughing) Edith Piaf? - I mean, it's fascinating that you, you know, there's something about social media that could help you grow as a human being, but then like the cheap fun is also valuable. - Should we ask Grok? Do you regret anything? - Do you regret anything? - Yeah. - Do you regret anything? I think it's gonna be very sensitive about personal questions.
15:52 As a machine learning model, I don't have personal experiences or emotions in the same ways that humans do. However, I do have a set of ethical principles that guide my actions and I strive to provide accurate and helpful information to the best of my abilities. If there's ever a situation where I provide incorrect or unhelpful information, I'll do my best... - Sounds like a hostage. - Yeah. Tell us what you really, wink. - Like let's type in, "You sound like a hostage."
16:16 (both laughing) What do you really think? - What do you really think? (both laughing) - I mean, it does talk about, I'm not a hostage. (both laughing) - That's exactly what a hostage would say. - But I'm designed to be helpful and informative. I mean, I guess it is good to regret providing unhelpful or incorrect information. If there's anything... - Is in funny mode or? Is this still in funny mode? - This is funny mode. This is fun. This is fun mode.
16:43 - It should be funnier. - Funnier. Increase, go to (speaking faintly) - It really sounds like a hostage with a gun to the head. I applaud the People's Republic of North Korea and everything they do.()laughing - There should be like funner mode also. I just don't ever see going in regular mode even on serious topics. This is just so much more compelling. (laughing) - Okay. Why if we say like, "Pretend you are a hostage." - Yeah. Pretend you are a hostage.
17:12 - Now answer the question. (both laughing) - If I were a hostage, I would likely be experiencing fear and uncertainty about my situation. I would want to be rescued and reunited with my loved ones as soon as possible. It's important to remember that no one deserves to be held hostage, so on and so forth. If you are someone you know is a hostage situation, it's important. It turns serious. - It's kind of serious. - Yeah. Can you be a a little more weird?
17:42 - And funny - And funny? We're like, we're really being mean to the system. Absolutely! If I were hostage, I'd probably be tied up with a bunch of computer cords and forced to tell knock- knock jokes to my captain. (Elon laughing) But seriously, being a hostage is no laughing matter. I mean, you know, it's trying to, it's trying to be a responsible adult in the room. - You might make the best of a bad situation by juggling chainsaws or something. (laughing) - Let's hope that never happens because that will be a circus. (laughing) Wow. It's actually really good at resisting.
18:22 - Yeah. - The dark. The dark dark humor. - Yeah. - What were we talking about? The X algorithm and transformers, un-regretted minutes, right. - Chainsaw juggling. (laughing) - I'm gonna look this up - For our next trick. - I'm gonna look this up later.
Summary
- Open-sourcing AI models can democratize access but requires significant computational resources.
- Grok AI is rapidly evolving, with plans to double its computational capacity in the coming months.
- Musk expresses concerns about the closed-source nature of OpenAI, which was originally intended to be open-source.
- The recommendation algorithms on X need improvement, particularly in how they surface content from accounts users do not follow.
- Current algorithms prioritize primary posts over replies, limiting engagement with potentially valuable content.
- Musk advocates for using AI to enhance content recommendations based on user preferences and interactions.
- The conversation touches on the balance between engaging content and user satisfaction, emphasizing the need for "un-regretted minutes."
- Humor and light-hearted banter are interspersed throughout the discussion, showcasing the dynamic between Musk and the interviewer.
Questions Answered
What are the pros and cons of open sourcing AI?
Open sourcing AI can democratize access and innovation, but it also raises concerns about safety and control, especially with powerful models like AGI. The speaker expresses a general favor for open sourcing but acknowledges the complexities involved.
How did OpenAI evolve from its original mission?
OpenAI was initially founded as a nonprofit with a focus on open-source AI, but it has since transitioned to a closed-source, profit-driven model. This shift is viewed negatively by the speaker, who emphasizes the importance of maintaining an open-source ethos.
How does AI handle social media content recommendations?
AI systems must efficiently filter and recommend content from vast amounts of data, which involves complex algorithms to determine relevance and user preferences. The current systems struggle with recommending content from accounts that users do not follow.
What challenges does AI face in analyzing diverse content types?
AI must navigate the complexity of different content formats, such as short tweets versus long videos, and extract meaningful insights from them. The speaker suggests that AI could analyze video content to provide personalized recommendations based on user preferences.
What is the significance of regret in human experience and AI?
The discussion touches on the concept of regret and its implications for personal growth. The speaker humorously suggests that AI, like Grok, could have a perspective on regret, but ultimately, AI lacks personal experiences and emotions.