The superstar success of Elon Musk, the world’s richest man, with a fortune now estimated at $246 billion, poses quite a puzzle. How can someone whose wealth and fame rest on innovative technologies—electric cars and rocket launches—be such a doofus? Never mind his cursing at advertisers, getting high on a podcast, fostering white supremacists or challenging another peculiar rich guy to fistfights, the globe’s most famous technology mogul has spent much of the past two years blowing $44 billion on Twitter (now X) only to drive it into the ground.
And yet this same character remains critical to Ukraine’s defense against Russia, U.S. national security space launches and the American automotive industry’s hopes of withstanding a flood of cheap Chinese electric vehicles. His Dragon capsule will retrieve astronauts stranded on the International Space Station by his doddering space competitor, Boeing, NASA announced in August. This same genius crudely proposed impregnating the pop star Taylor Swift, in front of millions of people, following the second presidential debate in September. Again, how can someone be so essential, so rich and, at the same time, such a dingbat?
A look at history and economics suggests that Musk, now age 53, won his fortune from a “second-mover” advantage, well known in innovation scholarship. With regulation, marketing and basic principles established, a savvy second comer—think Facebook trampling MySpace or Google crushing Yahoo—can take over a burgeoning industry with a smart innovation pitched at a ripe moment. Musk also enjoyed a great deal of good fortune, otherwise known as handouts from Uncle Sam and less astute competitors.
The rest of the article fleshes out that last paragraph.
I wouldn’t use the words “doofus” or “dingbat” but, yes, he’s a bit strange. So?
As for the rest, OK. Am I an Elon-fanboy? No. That’s not how I am.
Does he mess up? Yes, what he's done with Twitter is not a success story. And no matter how much he blathers about free speech, free speech is not his game at all. He's egocentric. I think it’s incredible. Still, look at what he’s done.
One of my main complaints about the current regime in machine learning is that researchers don’t seem to have given much thought to the nature of language and cognition independent from the more or less immediate requirements of crafting their models. There is a large, rich, and diverse literature on language, semantics, and cognition going back over a half century. It’s often conflicting and thus far from consensus, but it’s not empty. The ML research community seems uninterested in it. I’ve likened this to a whaling voyage captained by a man who knows all about ships and little about whales.
As a symptom of this, I offer this video clip from a 2023 conversation between Ilya Sutskever and Dwarkesh Patel in which next-token prediction will be able to surpass human performance:
Here's a transcription:
I challenge the claim that next-token prediction cannot surpass human performance. On the surface, it looks like it cannot. It looks like if you just learn to imitate, to predict what people do, it means that you can only copy people. But here is a counter argument for why it might not be quite so. If your base neural net is smart enough, you just ask it — What would a person with great insight, wisdom, and capability do? Maybe such a person doesn't exist, but there's a pretty good chance that the neural net will be able to extrapolate how such a person would behave. Do you see what I mean?
Dwarkesh Patel
Yes, although where would it get that sort of insight about what that person would do? If not from…
Ilya Sutskever
From the data of regular people. Because if you think about it, what does it mean to predict the next token well enough? It's actually a much deeper question than it seems. Predicting the next token well means that you understand the underlying reality that led to the creation of that token. It's not statistics. Like it is statistics but what is statistics? In order to understand those statistics to compress them, you need to understand what is it about the world that creates this set of statistics? And so then you say — Well, I have all those people. What is it about people that creates their behaviors? Well they have thoughts and their feelings, and they have ideas, and they do things in certain ways. All of those could be deduced from next-token prediction. And I'd argue that this should make it possible, not indefinitely but to a pretty decent degree to say — Well, can you guess what you'd do if you took a person with this characteristic and that characteristic? Like such a person doesn't exist but because you're so good at predicting the next token, you should still be able to guess what that person who would do. This hypothetical, imaginary person with far greater mental ability than the rest of us.
The argument is not clear. One thing Sutskever seems to be doing is aggregating the texts of ordinary people into the text of an imaginary “super” person that is the sum and synthesis what all those ordinary people have said. But all those ordinary individuals do not necessarily speak from the same point-of-view. There will be tensions and contradictions between them. The views of flat-earthers cannot be reconciled with those of standard astronomy. But this is not my main object. We can set it aside.
My problem comes with the Sutskever’s second paragraph, where he says, “Predicting the next token well means that you understand the underlying reality that led to the creation of that token.” From there he works his way through statistics to the thoughts and feelings of people producing the tokens. But Sutskever doesn’t distinguish between those thoughts and feelings and the world toward which those thoughts and feelings are directed. Those people are aware of the world, of the “underlying reality,” but that reality is not itself directly present in the language tokens they use to express their thoughts and feelings. The token string is the product of the interaction between language and cognition, on the one hand, and the world, on the other:
Sutskever seems to be conflating the cognitive and semantic structures inhering in the minds of the people who produce texts with the structure of the world itself. They are not at all the same thing. A statistical model produced through next-token prediction may well approximate the cognitive and semantic models of humans, but that’s all it can do. It has no access to the world in the way that the humans do. That underlying reality is not available to them.
What does Claude have to say about this?
I gave Claude 3.5 Sonnet Sutskever’s second paragraph and had a conversation about it. I wanted it to see if it could spot the problem. Claude saw problems, but in the end I had to tell it that Sutskever failed to distinguish between the structure of the world and the structure of the text.
For a lot of people black holes and string theory were topics that were filed in the mental box labelled ‘things I will never be able to get my head around”. However, all changed when Neil deGrasse Tyson began appearing on TV screens.
0:00 Intro
02:02 Early context
05:47 Your parents direct influence
12:39 Your father being racially abused
23:36 How to decide what I want to do with my life
26:52 What are you concerned about with the human race
30:05 Social media polarisation
42:40 Do we matter
47:48 Where does happiness and meaning come from?
54:46 Whats required for a happy life for you?
01:00:17 The perfect way to tell stories
01:13:39 What do you struggle with
01:17:32 Mental health
01:30:04 The last guest’s question
"during the generation of behavior, the brain functions like a heterarchical system consisting of interconnected regions that flexibly influence each other through processes with different dynamics... at a particular timescale, their respective functions can even be redundant."
Investigating constrained behavior during a single point in time obscures the unique contributions of different brain regions.
Bidirectional connections enable flexible interactions between regions, providing a substrate for partial functional redundancy.
Redundancy can be expressed in several ways, including several structures subserving similar functions at a point in time.
Theories and experiments that integrate across behaviors and timescales may help untangle unique functional contributions.
Abstract
The nervous system evolved to enable navigation throughout the environment in the pursuit of resources. Evolutionarily newer structures allowed increasingly complex adaptations but necessarily added redundancy. A dominant view of movement neuroscientists is that there is a one-to-one mapping between brain region and function. However, recent experimental data is hard to reconcile with the most conservative interpretation of this framework, suggesting a degree of functional redundancy during the performance of well-learned, constrained behaviors. This apparent redundancy likely stems from the bidirectional interactions between the various cortical and subcortical structures involved in motor control. We posit that these bidirectional connections enable flexible interactions across structures that change depending upon behavioral demands, such as during acquisition, execution or adaptation of a skill. Observing the system across both multiple actions and behavioral timescales can help isolate the functional contributions of individual structures, leading to an integrated understanding of the neural control of movement.
So far we’ve had 27 James Bond films, 25 by Eon Productions, and two others. I’ve seen many of them in theaters, perhaps even a majority, but certainly not all of them. Amazon has gathered the 25 Eon films together so we can conveniently binge them in chronological order or in whatever order we choose. In the past few weeks I’ve watched through them all, more or less in order.
When I started watching them I had no intention of publishing remarks about any of them, much less something about the lot of them. But then, as sometimes happens, I decided that perhaps I should write something. It happened like this:
I started watching from the beginning, with Dr. No (1962). This introduced much of the Bond nexus, such as the gun-barrel credit-opener, the women, the suits, the martini, the physical prowess, though not the gadget fetishism, which came two films later. It also introduced SPECTRE (Special Executive for Counter-intelligence, Terrorism, Revenge and Extortion) as a nebulous international criminal enterprise dedicated to enriching itself through elaborate schemes.
Then I watched the second one, From Russian with Love (1963). SPECTRE continues, and we get to see Robert Shaw as an athletic villain (13 years later he played Quint in Jaws). It became a smash success and was, and still is, regarded as one of the best Bond films. I read that judgment in the film’s Wikipedia entry, which I read after I saw the film (something I do regularly), and I concur. There’s minimal gadgetry and none of the big-tech nonsense which infested some of the Roger Moore films (I’m thinking of Moonraker in particular).
Next we have Goldfinger (1964). This may be the first Bond I saw in the theater, or was it From Russian with Love? I don’t recall, but whichever it was, the theater was packed. Auric Goldfinger, a dealer in gold bullion, is the villain. He’s a freelancer unaffiliated with any organization. This film introduced the first of the gadget tricked-out cars, the iconic Aston Martin DB5. Goldfinger also has an extensive pre-credit sequence which is unconnected with the rest of the film. That’s one of the features I had in mind when I started on this Bond orgy.
I was under the impression that that had become a feature of all the Bond films. Not quite. Most, perhaps all, later Bond films do have a long pre-credit sequence, but those sequences aren’t necessarily as disconnected from the plot as this one is. I suppose I should have been keeping notes on this, but I wasn’t, alas. In any event, this feature interests me because it’s characteristic of the Indiana Jones films and, in the case of The Temple of Doom, the precredit sequence is better than the film itself. The disconnected pre-credit sequence is also a feature of the delightful True Lies, which is somewhat of a parody of the spy-adventure film, perhaps even a comic critique of it.
That’s the first three Bond films, all with Sean Connery as Bond. He established Bond as a film character and was its best incarnation. If you’ve never seen any of the Bond films, watch these three, perhaps starting with From Russia with Love. After that I suggest you watch one of the Daniel Craig films (Skyfall, but I’m not sure it makes much difference), the most recent incarnation of Bond, and one of the Pierce Brosnan films (GoldenEye?), the penultimate Bond. After that, watch as you please. Roger Moore played Bond in seven films, more than any other. Some of the films were OK (perhaps Live and Let Die, with its Paul McCartney theme song), some were awful. He was the fourth Bond. Timothy Dalton played Bond in two films; he was the third Bond. George Lazenby was the second Bond, and had only one film.
It was a minor feature of that one film, On Her Majesty’s Secret Service (1969), that gave me the idea of writing something about Bond. The great Louis Armstrong shows up on the soundtrack singing a love ballad, “We Have All the Time in the World,” that was written for the film. It serves as the backdrop for Bond's courtship of the woman he rescues from suicide (Diana Rigg) in the opening segment. The song also appears over the end credits of the most recent (and likely the last) Bond film (he dies at the end), No Time to Die. There’s more to the film than Satchmo on the soundtrack and recent opinion ranks it as one of the best Bond films. I concur.
That leaves me with a last set of remarks. Bond kills a lot of people in these films, and he does it well. Others kill people too and many of them are expert at it. That is, it’s not just the killing, it’s the expertise in the craft. Why? I don’t think it’s sufficient to point out that, after all, death is a signal feature of human existence, one that’s the source of a great deal of anxiety and grief. How do we get from that fact to the appearance of death in so many of our stories?
And Bond, he’s not only expert in killing. He’s also a womanizer. Is that combination a mere contingency or is it central to the franchise’s appeal?
The thing about chess is that the structure of the search space is well-understood. The space may be unimaginably huge, but it's well-structured, and we know a great deal about searching such structures.
What's the search space for practical fusion power? Or the cure for pancreatic cancer? For the matter, what's the search space for the meaning of Shakespeare's Hamlet?
Search is only one aspect of a task, there are others – see my posts on chess and language. In general, what are the computational requirements of a task? If we want to find AI-shaped holes in organizations, we need to look at the tasks being performed and how those tasks are assigned to roles. In order to find AI-shaped holes we may have to refactor the way the computational requirements of tasks are assigned to roles so as to create AI-shaped roles.
I have found reading through these really valuable, and I haven't found an online source that compiles all of them in an easy to read format. So I made one.
Here's the first four emails:
Sam Altman to Elon Musk - May 25, 2015 9:10 PM
Been thinking a lot about whether it's possible to stop humanity from developing AI.
I think the answer is almost definitely not.
If it's going to happen anyway, it seems like it would be good for someone other than Google to do it first.
Any thoughts on whether it would be good for YC to start a Manhattan Project for AI? My sense is we could get many of the top ~50 to work on it, and we could structure it so that the tech belongs to the world via some sort of nonprofit but the people working on it get startup-like compensation if it works. Obviously we'd comply with/aggressively support all regulation.
Sam
Elon Musk to Sam Altman - May 25, 2015 11:09 PM
Probably worth a conversation
Sam Altman to Elon Musk - Jun 24, 2015 10:24 AM
The mission would be to create the first general AI and use it for individual empowerment—ie, the distributed version of the future that seems the safest. More generally, safety should be a first-class requirement.
I think we’d ideally start with a group of 7-10 people, and plan to expand from there. We have a nice extra building in Mountain View they can have.
I think for a governance structure, we should start with 5 people and I’d propose you, Bill Gates, Pierre Omidyar, Dustin Moskovitz, and me. The technology would be owned by the foundation and used “for the good of the world”, and in cases where it’s not obvious how that should be applied the 5 of us would decide. The researchers would have significant financial upside but it would be uncorrelated to what they build, which should eliminate some of the conflict (we’ll pay them a competitive salary and give them YC equity for the upside). We’d have an ongoing conversation about what work should be open-sourced and what shouldn’t. At some point we’d get someone to run the team, but he/she probably shouldn’t be on the governance board.
Will you be involved somehow in addition to just governance? I think that would be really helpful for getting work pointed in the right direction getting the best people to be part of it. Ideally you’d come by and talk to them about progress once a month or whatever. We generically call people involved in some limited way in YC “part-time partners” (we do that with Peter Thiel for example, though at this point he’s very involved) but we could call it whatever you want. Even if you can’t really spend time on it but can be publicly supportive, that would still probably be really helpful for recruiting.
I think the right plan with the regulation letter is to wait for this to get going and then I can just release it with a message like “now that we are doing this, I’ve been thinking a lot about what sort of constraints the world needs for safefy.” I’m happy to leave you off as a signatory. I also suspect that after it’s out more people will be willing to get behind it.
Sam
Elon Musk to Sam Altman - Jun 24, 2015 11:05 PM
Agree on all
There's much more at the link. Amazing stuff.
Reading through the rest I'm reminded of a line from The Blues Brothers: "We're on a mission from God."
Innovation is slowing, research productivity is declining, scientific work is becoming more [less] disruptive. In this video I summarize what we know about the problem and what possible causes have been proposed. I also explain why this matters so much to me.
A week or three ago I watched Rez Ball streaming on Netflix. It’s about basketball on the Navajo reservation (the rez), where basketball is close to a civic religion – think of football in West Texas as depicted in Friday Night Lights. It was directed by Sydney Freeland, a Navajo, and has an ensemble cast of people you never heard of because, well, it’s not that kind of film. But you may have heard of the producer, LeBron James.
Note, however, that the phrase “rez ball” also, and more fundamentally, indicates the style of basketball cultivated on the rez: “Run fast, shoot fast, don’t ever stop.” In the middle of the film we’ll see the Cuska Warriors playing against a seven-second shot clock which they sound out.
The film opens with credits playing on the screen while we hear voices commenting on some kids playing basketball. Then we zoom on in a desert while a voice utters, “From the shadow of beautiful Shiprock, coming at you with 50,000 watts” (as Shiprock, a recurring image, comes into view in the center of the frame). Now we see the kids, two of them playing one on one, with their parents commenting. One boy takes a three-point shot – “Jimmy with the three.” It bounces off the rim, and we jump cut to a third scene where a young man rebounds the ball. He takes it out and then drives in to make a layup. Finally, we see the title, “Rez Ball” over a black screen.
Obviously – at least in film logic – the two kids in the second scene have grown into the young men we see in the third. In the next minute or three we learn that one of the young men, Nataanii Jackson (the Braided Assassin), is a basketball star who sat out his junior year because his mother and sister (see saw them in the first scene) had been killed in a drunk driving accident. The other young man is his best friend, Jimmy Holiday (Kauchani Bratt), also a star player.
It's the beginning of a new season. The Cuska Warriors are about to play their first game of the season against the Gallup Bengals. They win, just barely. Their coach, Heather Hobbs (Jessica Matten), who’d been a star in college and then played professionally in the WNBA, is not happy – they’re ranked second in state – and lets them know it.
Their next game is against the Santa Fe Catholic Coyotes, who’d won the state championship the previous year. They crush the Warriors by 70. In the locker room after the game we learn why Nataanii Jackson hadn’t shown up to play. He’d killed himself.
At this point, roughly 25 minutes into the film, we know how the rest of the story will go, at least in rough outline. How do we know? It’s a sports film, and these films follow certain conventions. The Warriors will keep losing for a while, then they’ll get it together (a process that includes a day herding sheep), and come back to meet the Coyotes for the state championship. Do I have to tell you who wins that game?
The game is close at the half and stays close through the second half. Santa Fe is ahead, 78-77, as Holiday takes a shot at the final whistle. It doesn’t go in, but Holiday was fouled, so he gets three free throws. The first one goes in. Tie game.
Now the film switches gears.
I need to tell you about Holiday’s mother, Gloria Holiday (Julia Jones). She grew up with Coach Hobbs. They played ball together. Gloria was the better player, a natural. But...well, we’re not told, nor do the details matter. As the film opens she’s a single mom and an alcoholic. This and that, gets a job, starts going to AA meetings, gets a car, a junker. Decides to go to the championship game, which is off the rez. It’s one thing for her to drive on the rez, but once she gets off the rez she’s taking a risk because she has outstanding DUIs. Just as she arrives at the arena her car starts blowing smoke. That gets her pulled over by the police and then she’s taken to jail. But, knowing that her son is playing in the game, one of the officers sits down with her with a small radio and they listen to the final quarter.
We know all that as Jimmy gets ready to take his second shot. The lights go out, a spot comes on, shining directly on Jimmy over the basket from the left. The court disappears. A second spotlight shines over Jimmy from the right. We can see his mother sitting in a lawn chair: “That’s the thing about Natives. No matter how hard we try, we always find a way to lose. It’s in our blood.” He misses his second shot. We’re back in the court, fully lit. As Jimmy takes a couple of preparatory bounces the court once again goes black and his mother reappears: “The higher you go, the greater the fall. I just don’t wanna’ see you get hurt.” He replies, “Only one way to find out,” and takes the shot.
A day or two later Jimmy wakes up to hear someone shooting baskets in the hoop in front of his home. He goes out and sees his mother. They chat, she hands him a pile of recruitment letters, and remarks. “You gotta’ work on those free throws, though. Show me what you got, champ.” They start playing, talking trash.
Roll end credits.
* * * * *
While I don’t have a particular interest in sports films, I’ve watched a fair number of them. This is easily one of the best. Why? Think about those last two free throws and what the film had to do to support them.
This is a long, rambling, conversation (4 hours), so I have a hard time recommending the whole thing. I’d say that Wolfram and Yudkowsky do manage to find one another by the 4th hour (sections 6 & 7) and say some interesting things about computation and AI risk (much of the earlier conversation was on tangential matters). I note that the whole thing has been transcribed and there’s a Dropbox link for the conversation.
I will note that the conversation they did have was much better than what I had anticipated, which was a lot of talking past one another. And, yes, there was some of that, but as soon as that got going they worked hard at understanding what each was getting at.
Wolfram has some interesting remarks on computational irreducibility scattered throughout – that’s certainly one of his key concepts, and an important one. He also asserts, here and there, that he’s long been used to the idea that he faces computers smarter than he is; he also notes that he regards the universe as smarter than he is.
My sense is that the computation and AI risk stuff could be written up in a tight 2K words or so, but I don’t have any plans to make the attempt. That might be an exercise for a good student. Perhaps one of the current LLMs (Claude?) could do it.
TOC:
1. Foundational AI Concepts and Risks
[00:00:00] 1.1 AI Optimization and System Capabilities Debate
[00:06:46] 1.2 Computational Irreducibility and Intelligence Limitations
[00:20:09] 1.3 Existential Risk and Species Succession
[00:23:28] 1.4 Consciousness and Value Preservation in AI Systems
2. Ethics and Philosophy in AI
[00:33:24] 2.1 Moral Value of Human Consciousness vs. Computation
[00:36:30] 2.2 Ethics and Moral Philosophy Debate
[00:39:58] 2.3 Existential Risks and Digital Immortality
[00:43:30] 2.4 Consciousness and Personal Identity in Brain Emulation
3. Truth and Logic in AI Systems
[00:54:39] 3.1 AI Persuasion Ethics and Truth
[01:01:48] 3.2 Mathematical Truth and Logic in AI Systems
[01:11:29] 3.3 Universal Truth vs Personal Interpretation in Ethics and Mathematics
[01:14:43] 3.4 Quantum Mechanics and Fundamental Reality Debate
4. AI Capabilities and Constraints
[01:21:21] 4.1 AI Perception and Physical Laws
[01:28:33] 4.2 AI Capabilities and Computational Constraints
[01:34:59] 4.3 AI Motivation and Anthropomorphization Debate
[01:38:09] 4.4 Prediction vs Agency in AI Systems
5. AI System Architecture and Behavior
[01:44:47] 5.1 Computational Irreducibility and Probabilistic Prediction
[01:48:10] 5.2 Teleological vs Mechanistic Explanations of AI Behavior
[02:09:41] 5.3 Machine Learning as Assembly of Computational Components
[02:29:52] 5.4 AI Safety and Predictability in Complex Systems
6. Goal Optimization and Alignment
[02:50:30] 6.1 Goal Specification and Optimization Challenges in AI Systems
[02:58:31] 6.2 Intelligence, Computation, and Goal-Directed Behavior
[03:02:18] 6.3 Optimization Goals and Human Existential Risk
[03:08:49] 6.4 Emergent Goals and AI Alignment Challenges
7. AI Evolution and Risk Assessment
[03:19:44] 7.1 Inner Optimization and Mesa-Optimization Theory
[03:34:00] 7.2 Dynamic AI Goals and Extinction Risk Debate
[03:56:05] 7.3 AI Risk and Biological System Analogies
[04:09:37] 7.4 Expert Risk Assessments and Optimism vs Reality
8. Future Implications and Economics
[04:13:01] 8.1 Economic and Proliferation Considerations
An alternative view of what is happening is that we have been passing through three different phases of LLM-based development.
In Phase 1, "scaling is all you need" was the dominant view. As data, network size, and compute scaled, new capabilities (especially in-context learning) emerged. But each increment in performance required exponentially more data and compute.
In Phase 2, "scaling + external resources is all you need" became dominant. It started with RAG and toolformer, but has rapidly moved to include invoking python interpreters and external problem solvers (plan verifiers, wikipedia fact checking, etc.).
In Phase 3, "scaling + external resources + inference compute is all you need". I would characterize this as the realization that the LLM only provides part of what is needed for a complete cognitive system. OpenAI doesn't call it this, but we could view o1 as adopting the impasse mechanism of SOAR-style architectures. If the LLM has high uncertainty after a single forward pass through the model, it decides to conduct some form of forward search combined with answer checking/verification to find the right answer. In SOAR, this generates a new chunk in memory, and perhaps in OpenAI, they will salt this away as a new training example for periodic retraining. The cognitive architecture community has a mature understanding of the components of the human cognitive architecture and how they work together to achieve human general intelligence. In my view, they give us the best operational definition of AGI. If they are correct, then building a cognitive architecture by combining LLMs with the other mechanisms of existing cognitive architectures is likely to produce "AGI" systems with capabilities close to human cognitive capabilities.
