In the beginning: OpenAI Email Archives (from Musk v. Altman)

Habryka over at LessWrong:

As part of the court case between Elon Musk and Sam Altman, a substantial number of emails between Elon, Sam Altman, Ilya Sutskever, and Greg Brockman have been released.

I have found reading through these really valuable, and I haven't found an online source that compiles all of them in an easy to read format. So I made one.

Here's the first four emails:

Sam Altman to Elon Musk - May 25, 2015 9:10 PM

Been thinking a lot about whether it's possible to stop humanity from developing AI.

I think the answer is almost definitely not.

If it's going to happen anyway, it seems like it would be good for someone other than Google to do it first.

Any thoughts on whether it would be good for YC to start a Manhattan Project for AI? My sense is we could get many of the top ~50 to work on it, and we could structure it so that the tech belongs to the world via some sort of nonprofit but the people working on it get startup-like compensation if it works. Obviously we'd comply with/aggressively support all regulation.

Sam

Elon Musk to Sam Altman - May 25, 2015 11:09 PM

Probably worth a conversation

Sam Altman to Elon Musk - Jun 24, 2015 10:24 AM

The mission would be to create the first general AI and use it for individual empowerment—ie, the distributed version of the future that seems the safest. More generally, safety should be a first-class requirement.

I think we’d ideally start with a group of 7-10 people, and plan to expand from there. We have a nice extra building in Mountain View they can have.

I think for a governance structure, we should start with 5 people and I’d propose you, Bill Gates, Pierre Omidyar, Dustin Moskovitz, and me. The technology would be owned by the foundation and used “for the good of the world”, and in cases where it’s not obvious how that should be applied the 5 of us would decide. The researchers would have significant financial upside but it would be uncorrelated to what they build, which should eliminate some of the conflict (we’ll pay them a competitive salary and give them YC equity for the upside). We’d have an ongoing conversation about what work should be open-sourced and what shouldn’t. At some point we’d get someone to run the team, but he/she probably shouldn’t be on the governance board.

Will you be involved somehow in addition to just governance? I think that would be really helpful for getting work pointed in the right direction getting the best people to be part of it. Ideally you’d come by and talk to them about progress once a month or whatever. We generically call people involved in some limited way in YC “part-time partners” (we do that with Peter Thiel for example, though at this point he’s very involved) but we could call it whatever you want. Even if you can’t really spend time on it but can be publicly supportive, that would still probably be really helpful for recruiting.

I think the right plan with the regulation letter is to wait for this to get going and then I can just release it with a message like “now that we are doing this, I’ve been thinking a lot about what sort of constraints the world needs for safefy.” I’m happy to leave you off as a signatory. I also suspect that after it’s out more people will be willing to get behind it.

Sam

Elon Musk to Sam Altman - Jun 24, 2015 11:05 PM

Agree on all

There's much more at the link. Amazing stuff. 

Reading through the rest I'm reminded of a line from The Blues Brothers: "We're on a mission from God."

Hossenfelder on stagnation: “Science is in trouble and it worries me”

Show notes:

Innovation is slowing, research productivity is declining, scientific work is becoming more [less] disruptive. In this video I summarize what we know about the problem and what possible causes have been proposed. I also explain why this matters so much to me.

00:00 Intro
00:33 Numbers [1]
06:33 Causes
10:32 Speculations [2]
16:25 Bullshit Research
22:06 Epilogue

I've given a fair amount of attention to this topic under the label, stagnation. I've also written a working paper: Stagnation and Beyond: Economic growth and the cost of knowledge in a complex world, July 25, 2019.

[1] Here she opens by citing results from Bloom et al. Are Ideas Getting Harder to Find? (2017), and other more recent papers. 

[2] Mentions Fast Grants, with clips showing Patrick Collison and Tyler Cowen.

Midtown from Hoboken

Rez Ball: Community, Alcoholism, and Basketball [Media Notes 141]

A week or three ago I watched Rez Ball streaming on Netflix. It’s about basketball on the Navajo reservation (the rez), where basketball is close to a civic religion – think of football in West Texas as depicted in Friday Night Lights. It was directed by Sydney Freeland, a Navajo, and has an ensemble cast of people you never heard of because, well, it’s not that kind of film. But you may have heard of the producer, LeBron James.

Note, however, that the phrase “rez ball” also, and more fundamentally, indicates the style of basketball cultivated on the rez: “Run fast, shoot fast, don’t ever stop.” In the middle of the film we’ll see the Cuska Warriors playing against a seven-second shot clock which they sound out.

The film opens with credits playing on the screen while we hear voices commenting on some kids playing basketball. Then we zoom on in a desert while a voice utters, “From the shadow of beautiful Shiprock, coming at you with 50,000 watts” (as Shiprock, a recurring image, comes into view in the center of the frame). Now we see the kids, two of them playing one on one, with their parents commenting. One boy takes a three-point shot – “Jimmy with the three.” It bounces off the rim, and we jump cut to a third scene where a young man rebounds the ball. He takes it out and then drives in to make a layup. Finally, we see the title, “Rez Ball” over a black screen.

Obviously – at least in film logic – the two kids in the second scene have grown into the young men we see in the third. In the next minute or three we learn that one of the young men, Nataanii Jackson (the Braided Assassin), is a basketball star who sat out his junior year because his mother and sister (see saw them in the first scene) had been killed in a drunk driving accident. The other young man is his best friend, Jimmy Holiday (Kauchani Bratt), also a star player.

It's the beginning of a new season. The Cuska Warriors are about to play their first game of the season against the Gallup Bengals. They win, just barely. Their coach, Heather Hobbs (Jessica Matten), who’d been a star in college and then played professionally in the WNBA, is not happy – they’re ranked second in state – and lets them know it.

Their next game is against the Santa Fe Catholic Coyotes, who’d won the state championship the previous year. They crush the Warriors by 70. In the locker room after the game we learn why Nataanii Jackson hadn’t shown up to play. He’d killed himself.

At this point, roughly 25 minutes into the film, we know how the rest of the story will go, at least in rough outline. How do we know? It’s a sports film, and these films follow certain conventions. The Warriors will keep losing for a while, then they’ll get it together (a process that includes a day herding sheep), and come back to meet the Coyotes for the state championship. Do I have to tell you who wins that game?

The game is close at the half and stays close through the second half.  Santa Fe is ahead, 78-77, as Holiday takes a shot at the final whistle. It doesn’t go in, but Holiday was fouled, so he gets three free throws. The first one goes in. Tie game.

Now the film switches gears.

I need to tell you about Holiday’s mother, Gloria Holiday (Julia Jones). She grew up with Coach Hobbs. They played ball together. Gloria was the better player, a natural. But...well, we’re not told, nor do the details matter. As the film opens she’s a single mom and an alcoholic. This and that, gets a job, starts going to AA meetings, gets a car, a junker. Decides to go to the championship game, which is off the rez. It’s one thing for her to drive on the rez, but once she gets off the rez she’s taking a risk because she has outstanding DUIs. Just as she arrives at the arena her car starts blowing smoke. That gets her pulled over by the police and then she’s taken to jail. But, knowing that her son is playing in the game, one of the officers sits down with her with a small radio and they listen to the final quarter.

We know all that as Jimmy gets ready to take his second shot. The lights go out, a spot comes on, shining directly on Jimmy over the basket from the left. The court disappears. A second spotlight shines over Jimmy from the right. We can see his mother sitting in a lawn chair: “That’s the thing about Natives. No matter how hard we try, we always find a way to lose. It’s in our blood.” He misses his second shot. We’re back in the court, fully lit. As Jimmy takes a couple of preparatory bounces the court once again goes black and his mother reappears: “The higher you go, the greater the fall. I just don’t wanna’ see you get hurt.” He replies, “Only one way to find out,” and takes the shot.

A day or two later Jimmy wakes up to hear someone shooting baskets in the hoop in front of his home. He goes out and sees his mother. They chat, she hands him a pile of recruitment letters, and remarks. “You gotta’ work on those free throws, though. Show me what you got, champ.” They start playing, talking trash.

Roll end credits.

* * * * *

While I don’t have a particular interest in sports films, I’ve watched a fair number of them. This is easily one of the best. Why? Think about those last two free throws and what the film had to do to support them.

Zzhhh....(over exposure)

Yudkowsky + Wolfram on AI Risk [Machine Learning Street Talk]

This is a long, rambling, conversation (4 hours), so I have a hard time recommending the whole thing. I’d say that Wolfram and Yudkowsky do manage to find one another by the 4th hour (sections 6 & 7) and say some interesting things about computation and AI risk (much of the earlier conversation was on tangential matters). I note that the whole thing has been transcribed and there’s a Dropbox link for the conversation.

I will note that the conversation they did have was much better than what I had anticipated, which was a lot of talking past one another. And, yes, there was some of that, but as soon as that got going they worked hard at understanding what each was getting at.

Wolfram has some interesting remarks on computational irreducibility scattered throughout – that’s certainly one of his key concepts, and an important one. He also asserts, here and there, that he’s long been used to the idea that he faces computers smarter than he is; he also notes that he regards the universe as smarter than he is.

My sense is that the computation and AI risk stuff could be written up in a tight 2K words or so, but I don’t have any plans to make the attempt. That might be an exercise for a good student. Perhaps one of the current LLMs (Claude?) could do it.

TOC:

1. Foundational AI Concepts and Risks
[00:00:00] 1.1 AI Optimization and System Capabilities Debate
[00:06:46] 1.2 Computational Irreducibility and Intelligence Limitations
[00:20:09] 1.3 Existential Risk and Species Succession
[00:23:28] 1.4 Consciousness and Value Preservation in AI Systems

2. Ethics and Philosophy in AI
[00:33:24] 2.1 Moral Value of Human Consciousness vs. Computation
[00:36:30] 2.2 Ethics and Moral Philosophy Debate
[00:39:58] 2.3 Existential Risks and Digital Immortality
[00:43:30] 2.4 Consciousness and Personal Identity in Brain Emulation

3. Truth and Logic in AI Systems
[00:54:39] 3.1 AI Persuasion Ethics and Truth
[01:01:48] 3.2 Mathematical Truth and Logic in AI Systems
[01:11:29] 3.3 Universal Truth vs Personal Interpretation in Ethics and Mathematics
[01:14:43] 3.4 Quantum Mechanics and Fundamental Reality Debate

4. AI Capabilities and Constraints
[01:21:21] 4.1 AI Perception and Physical Laws
[01:28:33] 4.2 AI Capabilities and Computational Constraints
[01:34:59] 4.3 AI Motivation and Anthropomorphization Debate
[01:38:09] 4.4 Prediction vs Agency in AI Systems

5. AI System Architecture and Behavior
[01:44:47] 5.1 Computational Irreducibility and Probabilistic Prediction
[01:48:10] 5.2 Teleological vs Mechanistic Explanations of AI Behavior
[02:09:41] 5.3 Machine Learning as Assembly of Computational Components
[02:29:52] 5.4 AI Safety and Predictability in Complex Systems

6. Goal Optimization and Alignment
[02:50:30] 6.1 Goal Specification and Optimization Challenges in AI Systems
[02:58:31] 6.2 Intelligence, Computation, and Goal-Directed Behavior
[03:02:18] 6.3 Optimization Goals and Human Existential Risk
[03:08:49] 6.4 Emergent Goals and AI Alignment Challenges

7. AI Evolution and Risk Assessment
[03:19:44] 7.1 Inner Optimization and Mesa-Optimization Theory
[03:34:00] 7.2 Dynamic AI Goals and Extinction Risk Debate
[03:56:05] 7.3 AI Risk and Biological System Analogies
[04:09:37] 7.4 Expert Risk Assessments and Optimism vs Reality

8. Future Implications and Economics
[04:13:01] 8.1 Economic and Proliferation Considerations

Tom Dietrich on the current evolution of AI

Posted in a Substack conversation here:

An alternative view of what is happening is that we have been passing through three different phases of LLM-based development.

In Phase 1, "scaling is all you need" was the dominant view. As data, network size, and compute scaled, new capabilities (especially in-context learning) emerged. But each increment in performance required exponentially more data and compute.

In Phase 2, "scaling + external resources is all you need" became dominant. It started with RAG and toolformer, but has rapidly moved to include invoking python interpreters and external problem solvers (plan verifiers, wikipedia fact checking, etc.).

In Phase 3, "scaling + external resources + inference compute is all you need". I would characterize this as the realization that the LLM only provides part of what is needed for a complete cognitive system. OpenAI doesn't call it this, but we could view o1 as adopting the impasse mechanism of SOAR-style architectures. If the LLM has high uncertainty after a single forward pass through the model, it decides to conduct some form of forward search combined with answer checking/verification to find the right answer. In SOAR, this generates a new chunk in memory, and perhaps in OpenAI, they will salt this away as a new training example for periodic retraining. The cognitive architecture community has a mature understanding of the components of the human cognitive architecture and how they work together to achieve human general intelligence. In my view, they give us the best operational definition of AGI. If they are correct, then building a cognitive architecture by combining LLMs with the other mechanisms of existing cognitive architectures is likely to produce "AGI" systems with capabilities close to human cognitive capabilities.

An ancient scene from Jersey City

From Shakespeare to Trump @3QD

My latest post at 3 Quarks Daily: Shakespeare, the Starry Welkin, and Donald Trump. I’m playing around with the idea that, while we (whoever “we” are) regard Shakespeare as central to our culture, Elizabethan England is in fact a foreign culture. It may be in our cultural lineage, but it is still foreign. The language is somewhat different, as are the customs. The political system is quite different, a monarchy vs. a representative democracy. The economic system is quite different as well, though I don’t mention it in the post, nor do I mention the educational system.

I then work my way around to Henry IV, Part II, where I quote from the scene known as the rejection of Falstaff, which I’ve discussed in a number of posts at New Savanna, including, Trump, Gibbs & NCIS, and the Queen @3QD. This is the scene where Falstaff approaches his good body, Hal, who has now become Henry V and expects advancement at court. Hal rebuffs him, however, pointing out he is no longer the person he was when he was hanging out with Falstaff and the gang. Thus he distinguished between his personal life, with its duties and obligations, and his public life, as king. He may be the same biological individual, but he is a different social individual.

That’s an important distinction, one that Trump does not make. He has in the past, and presumably will do so in the future, treated his position as president as his personal fiefdom. That’s what corruption is about, failure to distinguish between one’s private personal life and one obligation to an organization in which one holds a position.

That’s one thing that was on my mind. But there’s another, and this is somewhat different. How has Shakespeare influenced our culture and society? On the one hand he has influence subsequent writers and so his influence has diffused through the population in that way. That process has been going on for over four centuries.

But what about his current influence? How does that work? The plays continue to be taught in secondary schools and in colleges and universities, and they continue to be performed in various amateur and professional venues. There is a considerable scholarly establishment devoted to studying his work and publishing about them. There is a sizeable institutional structure built around him – do a search on “Shakespeare festival” and see what comes up. How does that influence our culture and society? Much of that seems to me ceremonial and symbolic in nature (I include the scholarship). There’s something going on here that’s more than ordinary influence. I’m not at all sure how that works.

More later.

Mixed bag

This election is different

Ezra Klein, There’s Something Very Different About Harris vs. Trump, NYTimes, Nov. 3, 2024.

To Democrats, the institutions that govern American life, though flawed and sometimes captured by moneyed interests, are fundamentally trustworthy. They are repositories of knowledge and expertise, staffed by people who do the best work they can, and they need to be protected and preserved.

The Trumpist coalition sees something quite different: an archipelago of interconnected strongholds of leftist power that stretch from the government to the universities to the media and, increasingly, big business and even the military. This network is sometimes called the Cathedral and sometimes called the Regime; Trump refers to part of it as the Deep State, Vivek Ramaswamy calls the corporate side “Woke Inc.” and JD Vance has described it as a grave threat to democracy.

There's more at the link.

Life inside a Japanese prison

Witness

91.7 K subscribers

What's Life Like Inside A Japanese Prison? | Witness | HD Japan Jail Crime Documentary

From the YouTube page:

We gained unprecedented filming access to two Japanese prisons to find out if accusations that the system is inhumane are true.

What we witnessed was staggering. Inmates must march to their worksites attached by a cord; they are not allowed to look the guards in the eye; outside of scheduled leisure hours, they must maintain absolute silence, unless they have obtained prior permission to speak.

The treatment of suspects in custody pushes many to confess to crimes which they did not commit, as was the case with one man who spent 46 years on death row. He was finally exonerated six years ago, but was left broken.

We try to explain why a country which operates on strict principles of balance and order might choose such a repressive system, and see if this may explain Japan having one of the lowest crime rates in the world.

This documentary was produced by Nova Prod and directed by Marie Linton. It was first released in 2020.

The documentary talks about, and shows footage of, Carlos Ghosn. From his Wikipedia entry:

...a businessman and former automotive executive. He was the Chief Executive Office (CEO) of Michelin North America, chairman and CEO of Renault, chairman of AvtoVAZ, chairman and CEO of Nissan, and chairman of Mitsubishi Motors.

In 2018, he was arrested in Japan on suspicion of financial misconduct at Nissan, having been accused of understating his annual salary and misusing company funds. In 2019, while under house arrest awaiting trial, he escaped from Japan by concealing himself inside a large box which was shipped as freight on a private jet.

A Girardian Note about Spartacus [Media Notes 140]

I saw Spartacus when it came out in 1960. The movie led me to the book, which I read quickly. I believe I stayed up reading it most of a night, but I don’t remember whether or not I finished it at that time. I also bought the soundtrack album.

As you may know, the novel and film are about a slave revolt in ancient Rome in the first century BC. Did I know that back then? I don’t know. Does that matter? Probably not. But it matters for my current purposes. It matters because the most Girardian aspect of the film/novel is not in the historical record. Howard Fast invented it.

Let me explain.

Early in the film we see gladiators training. These men are slaves. They are engaging in exercises where they make the same moves. Are they imitating one another, or a model that we’ve not seen? I’m not sure it matters much, for, whatever the model, the imitation is deliberate and conscious. Girardian mimesis is unconscious.

A bit later Spartacus – based on a real person – attacks the trainer. Others join in, they overpower the guards, and escape the ludus, the gladiatorial school. They rove the countryside; other slaves join them. In time their force becomes quite large, with thousands and tens of thousands joining them.

The Roman Senate sends a force against the rebels. The force is defeated. The rebellion gathers strength. There is much politicking in Rome and Marcus Licinius Crassus is made First Consul and given charge of the Roman army. He defeats the slaves. This leads to a scene where Crassus addresses the defeated slaves. While Spartacus is directly in front of him, Crassus doesn’t know that. He informs the slave that they will not be put to death, but be allowed to live, albeit as slaves, provided either that Spartacus identifies himself or that they identify him. Spartacus is about to identify himself when his friend, Antoninus, quickly asserts “I’m Spartacus.” The other men quickly join in, each proclaiming himself to be Spartacus.

Imitation? Yes, but deliberate, conscious imitation. Nor does it lead them into conflict with one another. It is an expression of solidarity.

And it drives Crassus nuts. Meanwhile Crassus had found Varinia, Spartacus’ wife, on the field after the battle. He takes her into his household and attempts, unsuccessfully, to seduce her. She rejects him.

Now THAT’s Girardian mimetic desire. Crassus doesn’t even know which of those defeated slaves (dead or alive) is Spartacus. He only knows that Spartacus led them, and that this woman was his wife. Crassus desires Varinia (only) because she was wife to this rebel leader. His desire is sparked by an idea in his mind, not by envy for a concrete human being whom he identifies. 

Note that Varinia is not in the historical record. Howard Fast invented her when he wrote the novel in 1951. It's as though he added a Girardian critique, or at least a comment, to the existing historical record.

There is, of course, much more to the film that I’ve included in this sketch.

Upstairs

A.I. in Hollywood, Wait and See

Devin Gordon, What if A.I. Is Actually Good for Hollywood? NYTimes Magazine, Nov. 1, 2024. From the middle of the article:

Over several months of talking to people around Hollywood about A.I., I noticed a pattern: The people who knew the least about its potential uses in the filmmaking process feared it the most; and the people who understood it best, who had actually worked with it, harbored the most faith in the resilience of human creativity, as well as the most skepticism about generative A.I.’s ever supplanting it. There was a broad consensus about the urgency of confronting its many potential misuses — tech companies’ skirting copyright laws and scraping proprietary content to train their machine-learning models; actors’ likenesses being appropriated without their permission; studios’ circumventing contractual terms designed to ensure that everything we see onscreen gets written by an actual human being. I must’ve heard the phrase “proper guardrails” at least a dozen times. But as the prolific Emmy-winning television director Paris Barclay, who has six episodes of multiple shows airing this fall alone, put it, “That’s what unions are for.”

A bit later:

Then in late August, the California State Senate passed long-gestating, SAG-supported legislation requiring estate consent for A.I.-generated replicas of dead performers.

When I asked one writer-director about the practice, he didn’t even let me finish the question. “Nope, nope, nope, nope,” said Billy Ray, who wrote “Captain Phillips” (2013) and co-wrote the 2012 big-screen adaptation of “The Hunger Games,” and who spent his time during the strike hosting a studio-lambasting podcast. “It’s completely insincere, dishonest filmmaking. It’s a lie.” The counterargument I kept hearing, from artists and from technologists, is that filmmaking is a grand illusion at its core, and we all consent to being tricked — we’re paying to be tricked — when we walk into the theater or turn our phone sideways.

There's much more at the link.