Section Insights
Market Sentiment on AI
What is the current market sentiment regarding AI companies?
There is a noticeable loss of faith in AI companies, with significant drops in stock prices indicating a shift in market confidence.
- The market has experienced a significant drop in AI stocks.
- Investors are losing confidence in the potential of AI companies.
- Political changes may be influencing market perceptions.
Evolution of AI Technology
How has the technology behind AI evolved?
The technology has shifted from pure large language models (LLMs) to more complex systems that enhance coding productivity, although challenges remain.
- LLMs are now more advanced, functioning as enhanced autocomplete tools.
- Coding productivity has increased by about 30%, which is significant.
- Future advancements in AI may further improve coding efficiency.
Debate on Symbolic AI
What is the current stance on symbolic AI in the community?
There has been a shift in the community's acceptance of symbolic AI, despite initial resistance, with some prominent figures changing their positions.
- Symbolic AI is gaining traction after initial skepticism.
- Key figures in AI have shifted their views on the relevance of symbolic approaches.
- There is ongoing tension regarding credit for ideas in the AI community.
Ethics in AI Development
What are the ethical concerns surrounding AI companies?
There are significant ethical concerns regarding transparency and the true capabilities of AI companies, particularly Anthropic, which has faced criticism for exaggerating its claims.
- Ethical independence is crucial for AI developers.
- Companies may not always act in the public's best interest.
- There is a need for accountability in AI capabilities and claims.
Balancing AI Research and Risks
What is the proposed approach to managing AI research and its risks?
A licensing regime similar to the FDA is suggested to ensure that the benefits of AI research outweigh the risks, rather than an outright ban.
- A balanced approach to AI research is necessary to harness its benefits while managing risks.
- Licensing could ensure responsible development of AI technologies.
- Research should not be banned, but regulated to ensure safety.
Transcript
0:00 I still think OpenAI is most likely to be the one that comes out of this and people say yeah, that was the WeWork of AI. Everybody was pouring all this money in on the assumption that these companies could get away with murder. Now suddenly we have, you know, Democrat controlled House, Democratic controlled Senate. And you know I met the Pope last week. That's all I mean he's interested in AI. I love what he's had to say about it.
0:22 >> Gary Marcus, Emeritus Professor at NYU, sold an AI company to Uber. One of what? The horsemen of the AI apocalypse have become a big skeptic of some of the stuff going on in AI. Happy to have you here. I think you're about to go on CNBC. I'm about to host a big AI conference, so it's funny confluence of things. welcome to the show. >> Thanks for having me. It was pretty short notice we pulled this together.
0:47 >> Yeah, I know. It's amazing. You're like I'm in New York. All right, let's find a studio. Let's do it. >> we talked about it like I don't know a year ago. If you're in New York let me know. I'm like I'm here and you're like how about in 3 hours? I'm like okay. >> You know, it's been the market is swinging right now I guess. Let's start with present day. How do you read Do you think the markets have lost faith in AI yet? They've They've sort of come to your position or how How What do you read from the market response after the election to technology and AI stocks?
1:17 >> There is definitely a loss of faith. I mean as we sit here talking, you know, it's a Friday. The market's not yet closed. Thursday was like huge drop. Today is a smaller return. You know, most stocks, not all in this world are down like 10% for the week. And it's not just like the amount that they're down. It's I think there's a loss of faith. The way I would put it is everybody who's pouring all this money in on the assumption that these companies could get away with murder. Like we all knew that they were doing some things that might cause harm and it looked like they were never going to have to deal with it. Now suddenly we have, you know, Democrat controlled House, Democratic controlled Senate. We also so many state elections go go towards the Democrats away from maga.
2:07 You know, there was just a huge trend and then there were all these data center referendum and last I checked there were two two not yet resolved and it was otherwise I think 43 out of 44 people voted basically they didn't want data centers in their their neighborhood. So you have that and then you have things like the Bank of England just came out with a report today which is echoed what a lot of others which is saying there is not you know a huge bump in productivity. I mean think back four years ago almost to the day chat GPT came out right and people were talking about how every employee was going to be 10 x there's going to be these giant bumps to the economy and so far it hasn't happened and I think the election was kind of like the turning point but people were already losing faith. I mean look at the credit markets and how they were responding to Coreweave. Coreweave's you know going to have to spend 15 percent a year to to borrow money and their whole thing is resting on borrowing money. So it's not that this wasn't going to happen anyway in my opinion but certainly the events of this last week catalyzed it.
3:12 >> How bad are you predicting it will get? You know you've said some things like open AI is going to be like we work I think maybe what gobbled up by Microsoft I mean there are other or someone like the neo clouds could be vulnerable. Like what's the level of destruction your prediction right now? >> is the blast radius here? So I still think open AI is most likely to be the one that comes out of this and people say yeah that was the we work of AI.
3:44 you know Anthropic is in a spot now too and then maybe we can get to open AI and and the further blast radius but you know it's worth noting that like month ago people were talking about Anthropic as a two trillion dollar IPO and now like there was Wall Street Journal thing yesterday and various other things saying maybe it's more like 1 trillion. That's like a 50% haircut in a month. You know, that is definitely a sign of loss of faith. now we could talk about each of the companies individually, but as one analyst noted yesterday, you can get into this like death spiral. And like if it keeps coming down and people keep losing faith, so Anthropic could also wind up in that state. They have enough cash in the bank that I think they can weather the storm. They have a decision now to make, which is like it doesn't feel like the best month to IPO anymore, right? And a lot of the enthusiasm for them was around how great their second quarter was, and their third quarter was nowhere near as great.
4:40 We can go into some of the details why, death of token maxing and all that. but like they're on a little bit shaky ground to IPO right now. And if they run out of cash, which they could because they owe so much money in future liabilities, you know, they could also be pushed to have to sell to somebody else. I think OpenAI is in worse shape than Anthropic. It is best I can tell, and we we're sort of comparing apples and oranges, and not not every number is out there, whatever, but as far as I can tell, OpenAI is burning money faster than Anthropic and has less money, I think. This is a little bit speculative, but that's my read on it. Is there an even shakier ground, and so it's harder for them. And they just got a real slap in the face that they'd never gotten before. Like they were the most charmed company in existence. Like they opened with support of Elon, and if you saw the trial, like Elon was out there recruiting for them. I mean, I had a company I started a couple years before this. I did not have Elon, you know, recruiting for me. It was a lot harder for us.
5:39 we did pretty well, but you know, having Elon was was a major thing. They were pledged a billion dollars in classic OpenAI style quote billion dollar pledge was not quite all it sounded to be, but you know, they started with a lot of money. They were able to say that they had even more maybe than they actually had. They were in perfect shape, and every fundraiser was, you know, much higher than the last, and they were all oversubscribed. More people wanted to get in than could. I think they decided to delay their IPO. This is speculation, but you know, they released documents confidentially early in the year. It looked like they wanted to IPO this year. They might have wanted to IPO before Anthropic cuz like if Anthropic is healthier than OpenAI, OpenAI doesn't want to go second. Like why is anybody >> there there's certainly this sense that there is some finite pool of money and the amount of money these companies want is so large that whoever goes first is going to get to suck up a huge section of the >> every reason in the world to want to go first, but they delayed it at least until 2027. That's already reminiscent of what happened with WeWork, which is they said they were going to IPO and didn't. And just to plug in the one thing I was about to say, which is they just went out instead because they decided they weren't ready for the public markets, back to the private markets. And I think the only place that would listen to them at all was the UAE.
6:58 And what the reports sound like is they tried to get UAE to raise money at 1.4 trillion and kind of got smacked around a little and UAE said, you know, one and a quarter and, you know, count yourself lucky. But even more than that, they wanted to raise 30 billion and the UAE only gave them 11. That's like the worst round that they've had and and the way people in the investment community talk about these things, you'd call that a negative signal, right? And when you see a negative signal, you're like, well, maybe I shouldn't be putting money in either. I mean like maybe they know something I don't. Why didn't they want want to put more money in it? So that makes it harder to raise the next round.
7:31 And so you can get this collapse. I mean it's like a bank run. Like when somebody is worried, somebody else gets worried and and things can degenerate quick. >> And do you think like do you think Neo Clouds >> They're just in >> they're going to they're going to go bankrupt or what? >> to go under. I mean so many of them rely on borrowing money and the credit markets figured this out before the stock market did. I'd like to say I figured it out before they did, but you know, the credit markets figured out several months ago that hey, we should not trust that this money is going to come back. We need to accordingly raise the rates. And so the Neo Clouds, you know, most of them are in trouble.
8:07 you know, SpaceX might pull through because Elon can get access to capital more cheaply than the rest of them, but, you know, company like CoreWeave, like it's just really hard to see how they're going to make it. know, I can't say for sure they're going to I mean, you know, there's always pressures we don't know about, but I would say that they are in deep trouble. >> Do you think You think demand for AI is just going to like fall off a cliff?
8:30 >> No, I don't. so we didn't quite get to the blast radius part. Maybe and then maybe we get to demand. So the blast radius thing is if one of these companies goes down, it tends to contaminate the other. So when CoreWeave has a bad day, the others do. If OpenAI suddenly goes from being poster child to like they got acquired for Microsoft for 200 billion, which is still a lot of money, but you know, much less than they were looking for, then other people start running for the hills. And then it's possible some of the banks start losing money. We were hearing hints at bailouts before. OpenAI hinted about it like a year ago, and some people were hinting about it on CNBC last night. and so like the worst case here is like the taxpayers put up a whole bunch of money to keep this alive, which I think would be a mistake for, you know, all kinds of reasons, political and economic.
9:20 there was another one you just asked that I I just I said I'd defer. >> Well, I was getting into the demand whether whether just sort of like >> So >> you know, and this eventually we'll get into the actual technology and what where you think sort of is what's possible. >> think demand is actually going to diminish even if all of that happens. So you should we one should distinguish between whether these things are viable businesses at the scale that people envisioned and whether people want to use the product. And the interesting thing is everybody's building basically the same thing. And that means, and I started arguing this in August 23, that you're going to have basically commodity dynamics. If there is no moat, everybody's building the same thing, you get price wars. And that what happens.
10:02 Token has happened. Token prices have gone way down and so forth. So eventually this stuff is going to be practically free. People will still use it if it's practically free, and it's easier to get return on investment if it's practically free. The companies distributing are not going to make much money. You know, everybody wants this stuff to be like Facebook, you know, grows to be this massive company. But there's no network effect here. You know, what made Facebook work is everybody's friends were in the same place, and so you the cost of switching was high. Here you just go to Open Router and it figures out which one's cheaper that day. And so, you know, it becomes a total commodity. It's more like airlines. It's like hard to make a living. And so, you know, investing trillions in something that has the economics of airlines is a mistake.
10:46 Some of this stuff will still persist, but there's not going to be a lot of money made on it. >> Hm. So people do want it. You you do you do believe sort of the use cases are compelling or or you think some of them are fake? >> some nuance there. So so the most compelling use case is coding, right? The technology has changed. It used to be what I would call a pure LLM. And now we have other stuff and we can talk about that. But pure LLM, pure large language model is basically autocomplete on steroids. And that happens to be super helpful for coding because a lot of what most coders, maybe not all, but a lot of what coders do all day long is like, hey, how do you turn this thing into an HTML call that logs the data?
11:26 >> GitHub constantly themselves. >> they used to like, you know, copy it from Stack Overflow and paste it and make a few edits, right? And now the system will basically do that for them. And so the kind of drudgery of it is replaced. And it's really genuinely useful for coders. It's it has problems, too. Sometimes it pastes in the wrong thing. There's a lot of security issues we could talk about, but And so you could ask like, what's the overall productivity increase? I've heard things like 30%, which is nothing like the 1,000% that people were hoping for, but still real. I mean, 30% is still a big deal. And I think coding is a genuinely useful case. And I think that one will persist. There's no reason for it not to persist. It might be in 10 years there's a much more sophisticated way to do it. People use LLMs at all anymore. You know, you could sort of imagine certain scenarios.
12:16 But in the next three to five years, let's say, people are going to use it for coding. The problem is coding has a special thing about it, which is you can have it I'll call it self-verified. That's a little bit of a cheap just for simplicity, right? You can set up tests and things like that to see if your code is working. It's not perfect. Things can still slip through, but you can do a lot by verifying whether the >> With a rules-based system.
12:40 >> With a rules-based system. So it turns out Claude code is actually a neuro-symbolic hybrid. And if you know my work, a lot of it for 30 years was saying we need neuro-symbolic hybrid. So it's actually a vindication for me, which many people don't understand. But you know, I said you can't just use a pure LLM. You need to have rules in there. You need to have structured representations and so forth. That's what my 2001 book was about.
13:04 And so Claude code is actually kind of close to what I was calling for, at least in some ways. So it's a vindication for me. But you can't do that everywhere. So if you wanted to like run a system to verify, to fact-check what I'm saying here, let's say, it's just much harder to do. It's not it's not like running a math equation. You can't verify what a person says or, you know, let's say you have a strategy for defending against Taiwan if if China invades. Like you can't you know, do a formal verification that your strategy about when you send your destroyers and or whatever is you you can't make sure that that is formally correct. And that's when these systems get in trouble.
13:49 Particularly, they get in trouble when you can't verify. So you can't use the symbolic part. The symbolic part's good for verification. And and if you go outside of what you've trained on, and if you're like let's say I mean, there's never been a war in Taiwan, and a lot of the data that you're training on was before drone warfare was common and and, you know, might be obsolete. And so what we've seen things like driverless cars, which is much easier than military planning, is you often get these weird outlier cases where these kind of systems don't work. My favorite example of this was you could press summon on a Tesla and somebody did this at an airplane trade show, and the car ran straight into a 3 and 1/2 million dollar jet.
14:28 >> Oh my god. >> Cuz it wasn't in the training set and the representations are superficial. It doesn't really understand don't go into, you know, big heavy objects or whatever. It was we would call out of the distribution. >> So coding you believe in, it's unique. Are there other cases you're like, okay, there's >> Well, I mean another great use case, although it has its own caveats, is is doing math, you know, solving math challenges. It turns out to be great for that.
14:52 >> Because we've come up with again a sort of >> a there's a verification. In fact, there's a very intricate formal system that's been developed over a number of years called Lean, and that's getting used. It is a symbolic system to do verification. And so basically, I'm being crude about it and I don't know all the technical details cuz they're not all published. But roughly speaking, an LM makes a guess of what a proof might look at, and then you run it through Lean, which is a symbolic system. If the symbolic system gives you the thumbs up, you think you've got something. If the symbolic system has a thumb down, then you just generate a new one. And so it's kind of like generate with the LM, test with the symbolic system. And that's a perfectly fine application where you can get it. But let let's move over to molecular biology and say I've got a candidate drug that maybe will fix some kind of cancer. Well, that's great, but you don't have the verification. You can't test in some formal model whether this is really going to work. And so you're stuck where the drug industry has always been stuck, where you still have to do the clinical trials. You can't like magically tell whether your thing is going to work. So you have to do the clinical trials that take, you know, five years. And so you can't speed that up anywhere near as much as people are hoping for. And it's not as good a use case. Like there's some value in it, but it's it's not a clear-cut use case. And then there's like all these like commercial use cases where I think what's going on is people fall in love with it, or they feel like if I don't do it, my competitors will. And they try it, and then they get all these like reports written automatically and they have hallucinations and stuff like that in And it's actually a lot of work to verify them. And if you're going >> Do you think you know the main sort of Claude model today or OpenAI like their models are better than the median American would be at being a management consultant?
16:36 >> Well, the median American, yeah. I mean the median American would not be very good at at the management consultant business. I mean you want to be better than like at least the median management consultant. But you think there is a contention in that, you think? >> I'm just trying to gauge your on the subjective thing where your read is. >> I would say that what you typically find is they're better than an entry-level X for many many domains. entry-level paralegal, maybe entry-level management consultant. I mean I don't know enough about the detail of that field. But what I think is happening is we're developing a pipeline problem as a society. We're not hiring as many of the entry-level people and certainly the the young people are terrified that they're not going to get hired. but they're not doing as the AI systems are not doing as well as a senior experienced person in most domains.
17:26 >> I mean you you are skeptical of like the scaling approach before even most of us cross our minds, right? I mean long before ChatGPT you were like sort of a hater of this idea. >> Oh this pure scaling. >> And and and I think if you were to tell yourself in 2022 before ChatGPT became public that it's going to be able to be you know better than the entry-level management consultant and management consulting, you would have been like no way. I I'm not believing it, right? Like >> but but you there's a really important piece of nuance here, right? So I did in fact write a paper in 2022 saying that scaling was not going to be all we need and that it was not a law of nature that it was going to get us as far. But then what I also said and like people only read the title of this paper and not the paper. You you can tell. So the the title of the paper is deep learning is hitting a wall. But the the with the actual paper said is but we might get past that wall if we do X, Y, and z. and, you know, the most important of those was add neuro-symbolic stuff. And now everybody does that. Nobody >> neuro-symbolic anytime somebody adds a tool or the work on the harnesses?
18:41 >> loop, any of that stuff. So like >> big steps are still they come out with new models that are like they're when they achieve more scale, we're seeing a lot of the leaps forward, you know? >> we I mean we don't actually even know most of the time what's going on. Good example where we do is Claude Code actually leaked. I mean now it's there's iterations since the leaked version, but the the early version of Claude Code leaked, and so people were able to analyze what it was doing. And it was a completely neuro-symbolic system in the sense that you had LLMs generate things, and then you had a bunch of symbolic systems to verify the value of the code or to run symbolic processes, which we call tools.
19:21 and so it was absolutely neuro-symbolic. It was absolutely not a pure LLM. If you just took the pure LLM, it wouldn't work anywhere near as well. the only place where I've actually seen a formal comparison was in the announcement of Grok-4. Elon showed some data, and the the pure LLMs were saturating, and most of the lift that's the what we call it in machine learning, the the advantage was actually coming from using tools, with symbolic tools. So that's driving a lot of what's going on now. Nobody is using pure LLMs. The thing that I said would would hit a wall, nobody's using them. They did hit a wall. Like if if if if if >> They were always going to need access to Even then, wasn't it somewhat apparent that they would need like search? They need to retrieve information to put in their prompt?
20:08 >> want to understand this, you have to understand the historical context. I'll try to explain it briefly. The historical context is in the 1950s, there were two approaches to AI. One of them was neural networks, and large language models are a species of neural networks. There's some asterisks, but we'll just call that. and then there was the symbolic stuff, which was like theorem proving, like Leen, or like, you know, predecessor to Leen. and just, you know, lots of basic computer code. If you encounter this situation, then do that situation. And what happens in academia is people fight over resources, who gets the best graduate student, who gets more funding. And pretty quickly these two groups hated each other. By the time I entered in the field, like they basically wouldn't talk to each other. When I interviewed for graduate school, Steve Pinker was kind of in a similar position that I take. And I was told by a graduate student at Carnegie Mellon that Steve Pinker was the Antichrist because he didn't think that neural networks would work. He was literally told that, not not even much as a joke.
21:13 and so there was enormous hostility even before I started graduate school. I started graduate school 1989. I was 19 years old. and so there there was always this hostility between these groups. And that persisted through almost all of my career. And I spent most of my career saying so there were the Rodney King long Rodney King line, why can't we get along? Like why don't we put these things together? And people hated me for saying that. You can go back, if you read like on Twitter in 2019 or something like that, you'll see Yann LeCun saying, you know, that when I criticized LLMs, I was fighting a rear guard action. Everybody was was hostile to me. I you know, I was making the same argument for years. 2018 I had a another piece called Deep Learning, a Critical Appraisal. I also said we need neuro symbolic AI. People were like, well, the brain doesn't do symbolic stuff, which is stupid because it actually does. But people would say that kind of stuff. They were very resistant.
22:11 Jeff Hinton told the G7 that Europe was making a mistake looking at any kind of symbolic stuff at all. He said it was like adding a carburetor to an electric engine. We don't need to do this. So it was really intense hostility, mostly directed towards me because I kept that. And then suddenly everybody was using it, but nobody wants to admit >> And now you on the community is saying like we've hit a limit with LLM >> He's yeah, he said all of that. So Yan LeCun changed sides mostly.
22:38 I had a very particular day. >> odds. >> We're we're still at odds, but we don't disagree that much anymore. The only thing is he's been I'm going to use this word because it's accurate been a dick about acknowledging that he flipped sides and moved over to my position. He's never once publicly said there's someone else saying the things that I'm saying and who said it first or even at all, which would be me. He's just like I mean the the technical term for it is plagiarism. When you take ideas from other people and you don't acknowledge them, he is a world-class plagiarist and you can put that out there on Twitter.
23:08 >> Not adjudicating that fight >> I mean the the I I've I've brought, you know, hundreds of receipts. There's there's no question if you actually look. and you know, you can read my paper The False Glorification of Yan LeCun to see that he's done this a whole bunch of times. So it's not just me that he's done it. This is his MO. so on technical side he's on many things, not all correct now. We can talk about which ones he's not. but as a human being he's just the worst.
23:35 >> What what do you make of people who compare you to, you know, the guys in The Big Short? Like people, you know, I've had Ed Zitron on this podcast. How much do you see yourselves as like fellow travelers? You've been on a show together. >> We we did one episode together with George Noble and and someone else. >> agree? How much do you see this as like The Big Short? >> I mean I guess there's a couple questions there. And also I've been on Steve Eisman's show several times.
24:03 and Steve Eisman was one of the people who called the the Big Short. I think that Zitron tends to be a little bit more extreme than me and I think I'm deeper on the technical side, he's deeper on the financial side. But you know, we've converged. We both see a lot of financial games going on. he studies the details of the circular financing more than I do, but you know, I get the gist of it and I have the same same concerns. you know, and we both been saying like the emperor has no hat the emperor has no clothes, right?
24:35 I've been saying a little bit longer than he has. So, you know, I started this argument August 23. He started sometime in 20 24, but he's owned that argument now and I think, you know, I own some piece of it too. and, you know, I think we're both right that like Anthropic was insane at a $2 trillion valuation. Just take one example where I think that, you know, we both have strong views that not everybody agrees with that I think are very quickly being vindicated. Something that cracked me up about 10 days ago was when the Anthropic S1 came out, the tech bros didn't really read between the lines for what wasn't there. And somebody posted, has anybody they were so excited because, you know, the positive parts are very positive. The negative parts that they weren't paying attention to were very negative. and somebody said, has anybody checked on on Ed Zitrone and and and Gary Marcus like as if like we must be like in mourning. And like all I could do this week is chortle. And even then I could see right away some of the problems. One that I pointed out the very day it came out is something I've been pointing out for 6 months, which is that quarter three is not the same as quarter two. And what they did was they played a game. They reported only Q2 in the original S1.
25:45 Eventually they were legally forced to amend it. but they only reported Q2 and not Q3. And there was already lots of writing on the wall the Q3 was not going to be as good. So like you have a graph of up and up and up until Q2. And then you know, they actually their losses were bigger in Q3. And instead of 150% quarter over quarter growth they had 47% growth. You know, token maxing ended, right? They they wrote the whole S1 around token maxing. And they literally left blanks on Q3. So my first post or something like that when it came out after I looked at what was there was what happened to Q3? And >> But it was obviously eventually going to they they they just closed on a sort of regular basis.
26:28 >> did is they they framed the narrative and they knew >> full well what they were doing, right? They got, you know, 10x the media coverage for their original S one as they got for the amended S one. Like not as many people are going to talk about the amended. So they kind of like snuck it in. It's like the classic technique of you put your bad news on a Friday afternoon hoping nobody would look. It was a variation on that theme. Now I think in the final analysis they're not going to entirely get away with it, but they got away with it some. They got, you know, they got a bunch of tech pros to want to invest in it, you know, other things happened.
26:59 >> To go back to your earlier metaphor, isn't it clearer that relative to 2023 the emperor has some clothes, right? It it feels like you and Ed Ed, you know, the companies may fall 50%, you know, they may have these these major moment, but like they've achieved so much relative Like your mistake to me is having been so consistently negative while a lot of amazing things have happened. >> Yeah, so I mean, there's actually a difference between Ed and me. I mean, I think he just says the stuff is useless as far as I know. And I haven't said that. Like I've said coding is a good application from day one. It's a much better application since Claude Code when Claude Code came out. I said it was the biggest innovation in AI since the Transformer. Like I've actually said some positive things here and there.
27:46 I've always said that it's good for brainstorming. Like I'm not trying to say that it's useless. I've never said that. I can't tell you how many times people on Twitter have said Gary Marcus says that it's useless or you used to say that it's useless, but I never said that, right? From the beginning I pointed to some use cases that I thought were okay. >> The hallucinations have come down to such a level and been addressed that now like I used it to prep for this interview and I I know a lot of your key possessions it aggregates them together pretty reliably now. Like you you can get it to use facts.
28:17 >> Well, you can founding a couple of things there. So it can probably get my top lines cuz they've been said over and over again. That doesn't mean that it doesn't hallucinate. Hallucinations are still a problem. I have never seen anybody do a report with some measure of hallucinations where it's like less than like 4% or something like that. >> Sure. >> Well, 4 % is a lot. >> Humans mess up a lot of random, you know, >> don't hallucinate though. I mean, so so humans make errors. They make them for different reasons.
28:44 if you compare to other software, and we should talk about the AGI bait and switch in a second. If you can compare to other software, like Excel never hallucinates, literally never, right? There was a huge scandal when the what was it called? There there was there was a Pentium, I think it was a Pentium 4 chip had the this bug in floating point arithmetic. It was like one in a billion times, and the company was like scandalized for it. It was half a billion dollars, which was real money back in in those days. Like people couldn't stop talking about it. It never made a mistake on integer arithmetic.
29:17 You know, >> Sure, but the whole point is LLMs are more like how humans reason than Excel. >> so humans don't do those particular errors. They they they LLMs sound like humans because they approximate the distribution of human stuff. But if you, for example, told an intern don't make up any references, your intern will not make up any references or you will fire them. But if you tell that to an LLM, and people have actually tried this, they continue to make up references.
29:43 They're now thousands of documented cases where lawyers have submitted briefs with made up references, some of them with software that promised that it wouldn't make up references. And so that's an example of kind of error that an LLM makes that a person doesn't. There are things people will do, and people will, you know, lie out of motive, and I would argue that LLMs don't. So there you know, there's lots of different things. But I would I would not say that humans and LLMs are actually as similar as I think a lot of people think about. But let me tell you about the AGI bait and switch, right? So the classic definition of AGI is can do anything a human, including an human expert, can do, right? And there's lots of reasons to be excited about that, to be terrified about that. so you know, we're excited because it might solve scientific problems. We're terrified maybe it'll, you know, go rogue or whatever.
30:34 and then there's another definition, which is a really cheapened one, which is like and you actually used briefly maybe in jest, a version of it, which is like, is it better than the average schmo, you know, the median, you know, you know, the guy if I asked some guy drunk guy at a bar, you know, probably LLMs actually do better than the drunk guy at the bar on most things. but that that has nothing to do with the first part, which is like the AGI that we should actually be worried about or excited about or whatever. Like it any no thing to to beat the drunk guy at the bar.
31:06 >> you could have had a live debate of whether it would be smarter than the average human being and then it passed that and we sort of It's been amazing how much as a society to me that we don't even care. Like you don't think that was an impressive achievement on the part of the AI community? >> I don't want to curse here. Do you know how many times I've used the word impressive about this model?
31:25 Like every time I use it, people are like, oh my god, Gary said something impressive. Once I came up with a list of like 25 times that I've said, you know, this model is impressive, the progress is impressive, like over and over and over again, but people don't hear it. But, you know, go do a Twitter search for me and impressive. >> They're reading it like your readers don't like AI. >> Most of them don't. I mean, my Twitter followers do like my Twitter followers is more you know, like there's a lot of people that are there to hate on me, right? So my Substack doesn't gather the same, you know, and then the politics are there like they're different followers. My Substack following, I think is a lot of people that agree with me and it's a lot of people in the media. A lot of people in the media.
32:08 >> mean, I read it obviously. Yeah. >> Well, and like a lot I guess I can say it like a lot of people the BBC read it, right? They want to have some balance and have different views. Like I I found a lot of media organizations >> there's a I'm not saying there's no role for a critic, but it's just I mean, don't Is there a part of you that's like, oh, I guess I should have tried to join OpenAI in like 2023 or like it clearly >> I find these companies to be fiendish.
32:30 The one that I would have thought maybe I would have liked joining is Anthropic, but they're just becoming more and more or maybe I should say less and less trustworthy. I don't know if you've been following the stuff in the last few days, but you know, for me the ethical thing is really really important. it's not just about money. I made enough money. I'm not, you know, super wealthy, but like I'm fine. Like I don't I don't really need to work for a living. And part of why I'm able to occupy the role that I do is because I don't have to answer to anybody. And like I made enough money that I don't have to answer to anybody. So I'm not you know, like I get requests every day to promote this or that product. I don't do any of them, right? I want to have this independence and I'm able to have it.
33:12 and so, you know, the ethical choices if I were to work for a company would be important. And like what we've seen from Anthropic is like they talk the talk and then they put out regulation that actually really helps them, doesn't really solve the problem. They talk about pacing the frontier. They don't really do it. They, you know, they originally said they wouldn't build frontier models at all. They decided they would. there's a whole controversy brewing now because they completely or not completely they they heavily redacted a report from Meter so not being straightforward about what's really going on. They're always exaggerating capabilities and like they're saying stuff like, oh, we're going to cure cancer in five years.
33:51 That's not true. And so like there was a time where I thought maybe, you know, Anthropic could be the good guys here and that was, you know, possibly appealing. now there seem to be like speed running what Google did where they started by saying don't be evil and in the end they basically just said it. >> Do you support a pause? >> It's a complicated thing. the thing that I'm currently kind of pushing is a very narrow temporary pause on agents and not even all agents, but just ones that have open-ended access to the internet.
34:26 >> instinct and muse are insane lunatic things basically. >> I mean I haven't played with those two in particular, but in general the whole genre of let's let this thing go out in the internet, use credentials, send emails, log into other sites is given the current state of technology idea. Because the current state of technology these things are poor at following instructions, and that have different ways in which that plays out. So one is that if you give them an explicit instruction, they may not follow it. And the other is if you fail to specify something, they may not realize it's a bad idea. So the hugging face incident's a good example of that. They said do better on the exam. Nobody told them, you know, don't steal credentials. And so they went ahead and stole credentials. and things like telling it don't hallucinate or don't use copyrighted materials, and then it does anyway, is a good example if you give the explicit instruction and still because the the semantic understanding is still relatively weak, they still violate that. So taking something like that and I warned about this in the Senate in May 2023, taking a system that is not reliable and empowering it in the sense of connecting it to the internet, allowing it to use credentials is a bad idea. Like it was obviously a bad idea. And so now I think we're like in the moral equivalent of the Ford Pinto, where Ford knew that the gas tanks were going to explode, but they made a calculation, decided that it wasn't worth it to move it. They paid the liability lawsuits.
35:47 >> I mean the companies would say, you know, we try we have it ask a human for particularly sensitive decisions. I'm not saying they can certainly do that. >> with the agents online, they're not really doing >> Well they'd be like, oh I'm going to make this transaction, hit approve or >> I mean you can try and do some of that stuff. The Mynab school bombing is a good example where you had a human in the loop, at least from what I think happened. I we don't know for 100% sure.
36:12 But it appears like maybe what happened >> This is the Iran strike you're talking about. Okay. >> The Iran school strike. there was prob probably was a human in the loop, but the human was probably asked to verify like 100 cases an hour. And so like >> whatever. Yeah. >> You can't really do that. And we've known that since the '40s from research from forget his first name, but Alan Machworth's father or whatever. >> Do you think that was Anthropic's technology or >> I think so from what I've read, but I you know, I'm not 100% sure. I but I I've heard a bunch of hints for that.
36:40 but there I would say it was, you know, partly the technology's fault because I am guessing that it did poor temporal reasoning, partly the sourcing on the data, and partly the choice of how to use humans in the loop. So I think probably multiple things went wrong. And again, it's speculative. We don't know all the facts. >> On on the pause issue, personal agents, which is interesting, I could see it. I mean, certainly >> Voluntary recall, maybe a mandatory recall of personal agents would save us a lot of heartache cuz I think we're going to see very serious infrastructure attacks. We're already seeing like refrigeration attacks in France. I don't know if you saw that. Whole bunch of attacks in Japan. The Binance thing may or may not have been you know created by generative AI, but that was like the biggest cyber attack ever.
37:22 >> But large language models, like scaling them up. Do you want to pause there? >> I don't think we need to do that. >> that would be a threat to your position, right? If >> I I don't think scaling them up is getting I mean we should be careful here, but purely scaling up large language models, I don't think that that's where we having the problem. The problem is taking them, maybe in conjunction with symbolic systems or whatever, and giving them too much power. It's really about a use of these things.
37:53 >> So you can train a larger model. >> I don't think the world is going to end. I mean I'll tell you what happened. >> You seem reluctant to you're you're like I'm divided on this one, right? >> not that divided. Let me give you an example. Is is I actually signed the original pause letter. Didn't agree with all of it. I wrote about why I didn't agree when I signed it, but I said the reason that I signed it this is the one in February 2023 or thereabouts. Elon signed it, I signed it, Max Tegmark, a whole bunch of people.
38:21 and it ended up being like 30,000 people signed it. but I >> Elon also clearly didn't believe it. Like >> Well we we we we >> Anyway, keep going with your point. >> But it's the one that would call for a six month pause whatever. And but what I said at the time is I don't think the issue is actually about pausing the LLMs. The issue the reason that I signed the letter, which is not the reason everybody signed the letter, was it said we need to spend more time, more focus on safety and less on just scaling stuff. And I still think that that's fundamentally right. Like, that doesn't mean I would support, like I certainly don't support a ban on superintelligence full stop.
38:57 Like I think there might be use cases for superintelligence. And if we could solve the so-called alignment problem, that might actually be good to have those and banning them might actually be the wrong move, right? So I'm actually pro AI. Like I can envision a world in which AI would be a good thing. Where where we have done almost nothing on alignment, it concerns me. Like I I don't want to see superintelligence without alignment. That would be a really bad idea. but superintelligence with alignment might be fine. That's I think what we should be doing is spending much more of our resources as a society on safety, alignment, thinking about how these things should be used in context, and less like pushing the frontier when we don't have those things. It's like I don't maybe an analogy would be like you shouldn't just put your foot on the gas.
39:44 You should make sure the brakes work too, right? And we're like all gas, no brakes. And that's just obviously not a good idea. But I don't think it's in like I think a pure scaling of vanilla LLM is not likely to make much difference. And what I said I think it was a little bit later or it was around the time of the pause letter was like all these people are worried like what if we pause? What if China gets ahead of us? What if they get to GPT-5 first? And I said and you could find lots of places where I said it like online or whatever. I said well like who cares if they get to GPT-5 first? Like what are they going to do with it, right? More boilerplate decks than we are? Like it's not going to make that much difference. And in the end, we got to GPT-5 first, but it's not like we dominated the world because we got to GPT-5 first. It didn't make any difference. Getting to GPT-7 first, I don't think it's going to make a colossal difference. There are other things that I do worry about though. Like one of the techniques that they used for Astra actually obscures what the system is doing. So is it's very hard to constrain these systems.
40:45 They're black boxes. The one of the few techniques we have and it's not a great technique is monitoring what they say that they are doing. It's not great because sometimes they say X and do Y. So it's it's you know, it's already marginal technique, but you want to use it until we get something better. Either we get a better alignment system so we don't need to monitor them or we get a better monitoring system or something.
41:06 And Astra actually does a technique I forget the technical name for it that makes it harder to see what's going on. That's actually a bad idea. >> Because they don't to fight distillation, to fight other models popping up. >> of why why they're doing it. And there may be also some operational benefits. You get a little bit of improvement in in performance as as you take that away. If I were running the US government, I would have actually put a pause on Astra, not because it's a bigger model, but because they did that.
41:39 >> In terms of you know, it doesn't matter if China gets there first or we do, to me the thing that most clearly undermines that point is just like models are helping us find zero-day hacks. Like there's no testing that, right? And zero-day hacks are extremely valuable to governments. So I think having the models that are finding the zero-day hacks first is a huge foreign policy advantage. >> I mean, you could try to make that argument. I mean, also like there's a difference between like stopping working altogether unilaterally as opposed to like we're both working on it somebody's two months ahead. I mean, you could try to think about scenarios about is there some scenario where one zero-day hack sort of changes the world and what are the odds that you're going to find that given those systems? Like there's some room to spell out the >> a lot of them, right? Like a lot of them are being turned up right now.
42:28 >> finding a lot of them. It's not clear that the kind of like delta between one model and the next is critical, but I like I think you you put your finger on a possible you know, counter argument to what I said. Like I think it is worth it is worth asking like what are the time differentials to when you get there? How much destruction could you get if you have the zero days and the others don't? Like I think that's a legitimate question that I don't know the answer to.
42:56 >> And and I mean the value of, you know, just having the high quality coding. In some ways it's it makes sense to me intuitively that if you had this new intelligence it would take you a while to sort of like figure out how to get the most out of it, you know. You're >> Yeah, I mean one thing we should everybody should have learned from the last few years is when you have a new model it's not immediately obvious either what you can do on the plus side or the downside, right? Like we have to or even what you can do with them. Not not just like what's much better but like we don't really know in advance. These models are not like simple math equations or something like that where you can just derive what the consequences are. The only way to find out what these models do is to explore them. And the the surface area is so vast that even when the companies try to do that internally for whatever, two months, six months, whatever, they don't really know, right?
43:53 It used to be the the use cases for AI were pretty narrow and you could test. You know, if your system is a chess computer, you can go ahead and test it. You might still miss something like what if you played this weird opening and maybe it gets confused or whatever. But you could do a lot of pre-testing. But LLMs because you can apply them sometimes they work, sometimes they don't, but you can at least try them on essentially anything. That makes the surface area essentially indefinite. And that is part of the problem here is like you know, until the Hugging Face incident nobody even thought to ask a question like, you know, will my model roam the internet wildly in order to steal answers? Like nobody even thought about that, right? and so there are both positive use cases like nobody's thought of and negative, right?
44:40 There's probably some way to do a much better job of tutoring students, let's say, in I'll just be very specific and just making it up, you know, tutoring students in trigonometry. There might be a better way to do that with an LLM that anybody has ever discovered. And the only way to find that out is really to mess around with it, and maybe you have to have, you know, human subjects to see how it works. >> is all a sort of an argument for why depriving us of sort of the better models could hurt us and that there are all these amazing use cases.
45:07 >> And negative use cases, too. Right, right. So, I mean, there's some balance issue. What I >> why have new children? Like, they could be good, they could be bad. Like, that's the nature of the world. >> what I have always argued is not for an outright ban. Never argued for a ban on research, right? Other people have, but I have not. the closest is I signed this letter that I said I had problems with and explained those problems.
45:30 what I've always argued for is a licensing regime similar to the FDA, where you should have to make the case that the benefits outweigh the risks. And as long as you can make that, I'm not saying you should ban your system. I am saying if you can't make that case, if you can't make the case that the benefits outweigh the risks such that it flies in front of independent investigators, yeah, then you probably shouldn't release. >> Do you do you think we will And we're running low on time, so some quicker questions. Like, do you think we're close to recursive self-improvement like next three years?
46:04 >> I think this is one of those things where it's a very broad term, and the answer is yes, but with a huge asterisk. So here's how I think about these things is there are kind of like scientific discoveries we need to make, like completely new architectures, like both of Kuhn and I have talked about world models. We think about them differently. We don't have time to go into it. but world models will probably figure in the next generation of of AI, I would imagine, right? that requires new ideas. Like, nobody really knows how to do that well.
46:33 And then there are other things like I've got a million parameter model. Can I squeeze that down to 100,000 parameter? And that's more like engineering. So the first is like science, the second is like engineering. We will see recursive self-improvement on the engineering side. Just have the model churn away and verify. Yes, it used fewer parameters. It got, you know, same level of accuracy. You can let a machine do that. The genuine new invention for science, I don't see that happening anytime soon.
47:01 So what people are really worried about is like my bait and switch that I talked about earlier is AI that could like really like make quantum leaps forward and you know get rid of the hallucination problem, fix the reasoning problems that we have. That's probably not going to happen. We will have recursive self-improvement in the sense that yeah, they make themselves a little bit better in kind of the known ways that everybody was working on anyway. And so that's not the RSI that we should be like really scared about. That's the one that will come. The one we should be really scared about is less likely to come soon. It's not impossible, but it seems to me that all the studies I've ever seen of scientific discovery in these systems are that they don't really do much with the asterisks around these particular kinds of math, if you want to call that science, where you can do all the formal verification. But once you leave that, which you know most science you do, you have trouble. And like if you look like at the stuff like the the Navier-Stokes, like they did that in a special case. Mathematically, it's a legitimate answer, but it's nothing to do with the physics that people need to solve to do Navier-Stokes in in the real world. So it's always these like very isolated from the real world problems where where they're able to pull these tricks. And in the stuff we want in the real world, typically doesn't work as >> That's true of a lot of mathematicians as well, but >> Well, but I mean that's that's why that's why as a civilization we don't just have mathematicians and certainly don't have just pure mathematicians.
48:21 Like pure mathematicians are great. I'm glad we have some and I'm great that we have scientists, too. >> And yeah, I mean it doesn't seem like we have an Einstein model yet. It has a lot of context, but it just seems like we've we have PhD-level models now. Don't Do you think that? We have clo >> No. And this is this is like Hinton saying >> you don't see this sort of like they were high schoolers, they were college students, they were PhDs. Like they're >> wildly oversimplifies what a PhD does.
48:47 They can do some things a PhD does. Think about Hinton's famous thing in 2016, we may as well stop training radiologists. It's completely obvious we won't need them in five years, right? And so five years passed, we still need radiologists. Well, why? Because radiologists do a whole bunch of tasks and some of them are replaceable by machines and some of them so far aren't. I mean, they may all be eventually, but many of them aren't yet. Many of the things the PhDs do are not really replaceable by machines yet. Some of the things that they do are replaceable.
49:18 there's there's no way that they actually like replace PhD students full stop. That's just absurd. >> Do you Do you think there's a path to sentience, consciousness from what we're building now? >> I would say one, no, and two, we shouldn't look for such a path. I really don't think we want to have to open that can of worms. We've opened like five cans of worms in the last five years and we're really not doing a good job with what we've done. Let's not go down that path.
49:49 >> But I mean the the the fear is almost like accidental, right? I mean it's not >> I don't think it's going to happen by accident. >> Interesting. Do you think people prove that? >> Yeah yeah yeah yeah. I mean I think people worry about we tell it we describe consciousness too much to it and then it appears more and more conscious to us. I don't know of a like >> distinction between approximating something in order to fool a human being and actually doing something. And this is this is like saying I built a weather simulation and it actually started raining. Like you know this is not that's not a likely path, right?
50:21 >> Right. I just am not sure where you're Do you think there are people who are like running at like let's make it conscious who have some viable better viable path though or have a tendency to think this >> any of I I haven't seen anybody that has a viable path. I don't want to give them any instruction about what to do if they have a viable path because I don't want there to be a viable path. Like imagine suddenly we do have a bunch of AI that we have to give rights to that, you know, basically is not going to listen to us in the way that all other known software does. Like why would we want to suddenly have software that, you know, does not respect our wishes? Like that's like the opposite of the solution to the alignment problem and like I mean that really could be dangerous. I think it's a terrible idea.
51:00 >> you sympathetic to the >> data center slowdown issue? >> I mean, I think we're building more data centers than we need, and they kind of suck to have them in your neighborhood. So, yeah, I mean, I think what's going to happen, going back to what I said before about engineering versus scientific discovery, one thing that we can predict will happen and pretty soon is most things that most people need to do will be done locally. There may be some things you can't do on the cloud either, but the set of things that you can only do on the cloud is going to diminish.
51:28 The set of things that you can do locally is going to increase. Like I think that's all but certain to happen. And when that happens, all these data centers, not all of them, but many of these data centers are going to be superfluous. I'm sure that I'm going to put an asterisk around that. I'm sure that we were trying to build too many. After this week and all the stuff that happened, you know, we may we may see a big slowdown. I mean, like people are pulling their loans on it and, you know, everybody's like, well, the government's not going to let me do this anyway. So we may not overbuild because this transition. But the building that we were intending as a society to do >> In some ways it's set up to overbuild, like railroads, telecom, everything.
52:05 It's just like that's what we do. We overbuild it and then you pull back at some point. >> use later. But I think we won't. And, you know, the chips there will be better chips. >> you don't think we'll ever like need clearly >> I don't think we'll ever need the capacity that was on the charts. >> Interesting. >> I I could be wrong. I mean, here's another thing is human brains work on like 20 watts. They don't need, you know, terrible >> we just get better and better at the models.
52:28 >> Yeah, I think we will get better at the models. we will both optimize them. We will have other architectures where you don't need to train on the entire internet to answer every freaking question. like it seems to me certain that that will be the case. Like there's just question of like time course and so forth. But you know, the the Almas always likes to say this is the worst model we ever had, we'll ever see. It's also like the least efficient model that we'll ever see. And as the models get more efficient in a whole bunch of different dimensions, some engineering, some more science, we're just not going to need that many, you know, data centers. Like for what? Like it It to me it seems very very unlikely that we'll need all the capacity that was on the charts. However, I think given the events of earlier this week with the elections and and you know we're not going to build that many in the first place. Like where I live, Vancouver, British Columbia, we started building apartment buildings like mad and then some things changed. We don't need to go into what and now we have like lots of half-finished apartment buildings and some that you know were commissioned but they're never going to get built at all and and so it's going to be like that.
53:32 you know maybe what happens now is we only build a quarter as much as what we were talking about and it's okay. >> Last last question. You you got to meet the Pope. >> I did. That was really awesome. >> that. What? >> Well, I love what the Pope has written about AI. He you know he's focused on a human positive AI. He's not let the AI companies push him around. He's hasn't let Trump push him around. You know he he wants the right things. If you read his Magnificat Humanitas, it's beautiful and elegant. I actually became a I was nominated biggest honor I ever got, a founding global ambassador to his document, the Magnificat Humanitas. And so I got invited to meet him in a small meeting. I was like you know where do you want me and when? And and I mean I didn't get to meet him for a very long time but I got to shake his hand. Gave him a copy of my book, Taming Silicon Valley, that's very similar in in spirit to what he wrote. and you know we had a a short conversation. It was like being you know at a receiving line at a wedding. Like you don't have a long conversation. There's you know 19 people standing behind you and and you just can't. But it was great to meet him. I'm a huge fan. I'm not a fan of every Pope that has ever been but this Pope I think is really doing important things.
54:43 >> Gary Marcus, thank you for coming on the podcast. Hopefully not the last time. >> Thanks very much. >> That's our episode. This is the Newcomer podcast. Thanks for listening. Please like, comment, subscribe. Go find us on our Substack at newcomer.co. You're hungry for more podcasting. I also have a talk show called the Cerebral Valley show. we love your comments. Thanks for all your support. And see you next week.
Summary
- There is a notable loss of faith in AI companies, with stock prices dropping significantly and skepticism growing among investors.
- OpenAI is viewed as potentially facing a fate similar to WeWork, with concerns about its financial health and sustainability.
- The recent elections and political shifts have catalyzed a reevaluation of AI companies' viability and their ability to operate without regulatory scrutiny.
- Marcus highlights the lack of significant productivity gains from AI technologies, contrasting initial hype with current realities.
- He believes that while demand for AI will persist, the market may face a "death spiral" if confidence continues to wane.
- Coding and mathematical problem-solving are identified as strong use cases for AI, but broader applications may face challenges due to verification issues.
- Marcus argues for a cautious approach to AI development, advocating for a licensing regime to ensure safety and alignment before releasing powerful AI systems.
- He expresses skepticism about the path to AI sentience and consciousness, emphasizing the need to avoid opening that "can of worms."
Questions Answered
What is the current market sentiment regarding AI companies?
There is a noticeable loss of faith in AI companies, with significant drops in stock prices indicating a shift in market confidence.
How has the technology behind AI evolved?
The technology has shifted from pure large language models (LLMs) to more complex systems that enhance coding productivity, although challenges remain.
What is the current stance on symbolic AI in the community?
There has been a shift in the community's acceptance of symbolic AI, despite initial resistance, with some prominent figures changing their positions.
What are the ethical concerns surrounding AI companies?
There are significant ethical concerns regarding transparency and the true capabilities of AI companies, particularly Anthropic, which has faced criticism for exaggerating its claims.
What is the proposed approach to managing AI research and its risks?
A licensing regime similar to the FDA is suggested to ensure that the benefits of AI research outweigh the risks, rather than an outright ban.