Weeds & water

China Viewed from China | Robert Wright & Peter Hessler [US more militaristic than China]

Timestamps:

0:00 Peter’s new book, Other Rivers: A Chinese Education
3:00 Have Chinese people become less hostile to America?
12:48 What Americans get wrong about China
17:14 Why Peter thinks Covid didn’t come from a lab
20:35 China’s transformative recent decades
30:46 Respect for authority in Chinese culture
40:02 Change in China between Peter’s two teaching stints there
54:17 Why the Chinese political system may change dramatically

From late in the conversation (1:01:48)(autogenerated):

yeah and I think you know sometimes we're looking in the mirror a little more than we might realize right I mean so we tend to see China through the lens
of the military right and we have a very militaristic view of China now um but the truth is like the United States is a very militaristic Society um you know and and China is really not right I mean
it's not a this is not a societ I mean it's kind of you know I I talk about those kids I taught in the '90s who became middle class Urban I think I know
one whose kid has something he went to a military college but he's not in the military like you don't Aspire in China
to send your kid into the military it's nothing no Elite for a person who's like Highly Educated whatever want their kid
to do that right but in America I mean I could go to Prince University I knew lots of kids in Princeton who were going to go to the military you know it's a
way to rise in America it's a way to become politically powerful it really isn't the same in China like it's a very different type of society so that's
always been my view also on sort of the Taiwan issue so is like this has not been a society that for the last 40
years the military has not been driving their policy now it often drives our policy in the United States but it has
not been the pattern in China

Publisher's blurb for Other Rivers:

An intimate and revelatory account of two generations of students in China’s heartland, by an author who has observed the country’s tumultuous changes over the past quarter century

More than two decades after teaching English during the early part of China’s economic boom, an experience chronicled in his book River Town, Peter Hessler returned to Sichuan Province to instruct students from the next generation. At the same time, Hessler and his wife enrolled their twin daughters in a local state-run elementary school, where they were the only Westerners. Over the years, Hessler had kept in close contact with many of the people he had taught in the 1990s. By reconnecting with these individuals—members of China’s “Reform generation,” now in their forties—while teaching current undergrads, Hessler gained a unique perspective on China’s incredible transformation.

In 1996, when Hessler arrived in China, almost all of the people in his classroom were first-generation college students. They typically came from large rural families, and their parents, subsistence farmers, could offer little guidance as their children entered a brand-new world. By 2019, when Hessler arrived at Sichuan University, he found a very different China, as well as a new kind of student—an only child whose schooling was the object of intense focus from a much more ambitious cohort of parents. At Sichuan University, many young people had a sense of irony about the regime but mostly navigated its restrictions with equanimity, embracing the opportunities of China’s rise. But the pressures of extreme competition at scale can be grueling, even for much younger children—including Hessler’s own daughters, who gave him an intimate view into the experience at their local school.

In Peter Hessler’s hands, China’s education system is the perfect vehicle for examining the country’s past, present, and future, and what we can learn from it, for good and ill. At a time when anti-Chinese rhetoric in America has grown blunt and ugly, Other Rivers is a tremendous, essential gift, a work of enormous empathy that rejects cheap stereotypes and shows us China from the inside out and the bottom up. As both a window onto China and a mirror onto America, Other Rivers is a classic from a master of the form.

Mundane

Watching film increases empathic understanding of formerly incarcerated people

Rebecca Keegan, Can Watching Movies Rewire Your Brain? Hollywood Reporter, October 23, 2024.

Five years later, people are climbing into an MRI machine in the basement of Stanford’s psychology department to see how watching Just Mercy quite literally changes their brains, part of the first academic study using a specific cultural product to measure empathy.

The brain imaging research, which Eberhardt is conducting with fellow Stanford psychology professor Jamil Zaki, is still underway, but the first phase of the study, which relied on participants watching videos online, hints at the potential of a movie to change minds. According to findings published [PDF] Oct. 21 in the Proceedings of the National Academy of Sciences, watching Just Mercy increased participants’ empathy for the recently incarcerated and decreased their enthusiasm for the death penalty.

The study is a test of what psychologists call “narrative transportation,” the idea that when people lose themselves in a story, their attitudes change. It’s the academic version of the frequently shared Roger Ebert quote in which he called movies “a machine that generates empathy,” and it’s a notion that many who work in the entertainment industry assume to be true but that no one has measured in such a scientifically rigorous way until now.

There's more at the link.

Prospects for using LLMs in content moderation for social media

From the YouTube page:

Wherein I am joined by the estimable Dave Willner, who helped build the content moderation system at Meta, who talks me through how and why Facebook has squelched my open letter to former Russian Ambassador Anatoly Antonov, and why my appeal has fallen into a black hole--and how his scheme of using large language models to scale content moderation might do a better job.

The first part of the discussion concerns Ben Wittes' case at Facebook. Willner discusses his work on using LLMs in contentent moderation starting at about 40:30. You might want to start there if you already know something about the content moderation process or have been through it yourself. You can always loop back to the beginning if you're interested.

In the current regime a bit of content may be flagged in response to a user complaint but more likely will be flagged an automatic classification system. If an item of yours is flagged it will be removed and you'll be notified of that and given some general reason, such as violating community standards. As to just which community standard or standards and how your item violates it, you'll be told nothing. You'll also be given an opportunity to appeal. That appeal is likely to be reviewed by a human, but who know if and when that will actually happen.

In an LLM-enhanced content moderation regime, the LLM will be able to provide information about the standards being violated and provide fairly specific reasons why a particular bit of content has been flagged. Moreover the user could enter into a dialog with the system. If that doesn't resolve the issue, then the case could be passed on to a human for review.

This strikes me as a significant opportunity to improve social media. And it seems plausible. Two posts I did back in December of 2022 illustrate the kind of reasoning that would be involved: Abstract concepts and metalingual definition: Does ChatGPT understand justice and charity? (Dec. 16, 2022) – which is included in a working paper: Discursive Competence in ChatGPT, Part 1: Talking with Dragons, Version 2 (January 11, 2023), ChatGPT the legal beagle: Concepts, Citizens United, Constitutional Interpretation (December 27, 2022).

Hossenfelder: Wolfram's research program seems healthy (after all). Perhaps it can work.

From the webpage:

Mathematician and Computer Scientist Stephen Wolfram wants to do no less than revolutionizing physics. He wants to do it with computer code that gives rise to all the fundamental laws of nature that we know and like -- and maybe more. Unfortunately, Einstein’s theories of general relativity inherently clash with how computers work. And yet, he and his team might have found a clever way around this problem.

Douthat: Real AI is not like the AI of the movies [bewitched by language]

Ross Douthat, Our Robot Stories Haven’t Prepared Us for A.I., NYTimes, Oct. 26, 2024.

Data the android experiences existential angst because he is obviously a self that is having a humanlike encounter with the strange new worlds that the U.S.S. Enterprise is charged with exploring. Pinocchio has to learn to be a good boy before he becomes a real boy, but his quest for goodness presumes that his puppet self is already in some sense real and self-aware.

Yet that’s not how artificial intelligence is actually progressing. We are not generating machines and bots that exhibit self-awareness at the level of a human being but then struggle to understand our emotional and moral lives. Instead, we’re creating bots that we assume are not self-aware (allowing, yes, for the occasional Google engineer who says otherwise), whose answers to our questions and conversational scripts play out plausibly but without any kind of supervising consciousness.

But those bots have no difficulty whatsoever expressing human-seeming emotionality, inhabiting the roles of friends and lovers, presenting themselves as moral agents. Which means that to the casual user, Dany and all her peers are passing, with flying colors, the test of humanity that our popular culture has trained us to impose on robots. Indeed, in our interactions with them, they appear to be already well beyond where Data and Roz start out — already emotional and moral, already invested with some kind of freedom of thought and action, already potentially maternal or sexual or whatever else we want a fellow self to be.

Which seems like a problem for almost everyone who interacts with them in a sustained way, not just for souls like Sewell Setzer who show a special vulnerability.

No one is ready for the AIs we're currently creating, certainly not the people who've built then. Them are strange creatures. We're all confused, and groping. 

There's more at the link.

Saudi Arabia's plans 15 soccer stadiums for the 2034 World cup

11 stadiums are slated to be built from scratch while 4 existing stadiums will be renovated.

0:00 Kick-off
1:05 King Salman International Stadium
1:48 King Faud Sports City Stadium
2:30 Roshn Stadium
3:20 Prince Mohammed Bin Salman Stadium
4:05 BetterHelp
5:27 Jeddah Central Development Stadium
6:09 Prince Faisal Bin Fahd Sports City Stadium
6:27 King Saud University Stadium
6:45 New Murabba Stadium
7:25 NEOM Stadium
9:19 King Abdullah Economic City Stadium
9:27 King Abdullah Sports City Stadium
9:47 Qiddiya Coast Stadium
10:31 South Riyadh Stadium
10:51 Aramco Stadium
11:47 Human Rights in Construction
12:22 Extra-Time
13:16 Penalties

Assembly Theory and the Nature and Origins of Life [Star Talk]

From the webpage:

What is life? Neil deGrasse Tyson and co-host Chuck Nice tackle assembly theory, artificial life, and the origin of lifeforms in the universe as we revise the definition of life with astrobiologist and theoretical physicist Sara Imari Walker.

Is life the biological systems we know, or could life exist in ways we have yet to discover? Starting with the familiar—carbon-based life forms— we explore the idea that life may be a universal physical process governed by laws of physics we have yet to uncover.

Is life fundamentally tied to chemistry, or could it be explained by a new kind of physics? Learn about assembly theory and the line between chemistry and when something can be considered alive.

Walker explains how the Miller-Urey experiment sparked a quest to understand life’s origins, but that experiment alone wasn't enough to explain life's complexity. From prebiotic chemistry to evolutionary steps, the discussion teases the thin line between randomness and selection.

Could life be the result of a phase transition, like water turning into ice? We discuss the search for extraterrestrial life and possible alternative biochemistries. Could life exist on other planets with different chemical structures? And what role does complexity play in this grand equation?

As technology advances, can artificial intelligence be considered alive? Could it represent a new form of life, co-evolving alongside humans? Could the future hold life forms and technologies beyond our current imagination? What new laws of physics might emerge from our growing understanding of complexity, evolution, and information?

One thing is certain: the journey to discover life, in all its forms, is just beginning.

Timestamps:
00:00 - Introduction: Sara Imari Walker
5:20 - Updating The Definition of Life
10:10 - Miller-Urey & The Line Between Chemistry and Biology
15:13 - Creating Aliens
20:29 - The Traditional Definition of Life
25:03 - Testing Assembly Theory
29:48 - Is DNA Fundamental?
37:46 - Is AI Alive?
45:20 - Free Will
47:37 - Entropy
50:10 - Life as No One Knows It [Walker's book]
52:58 - A Cosmic Perspective

Just a reminder [white iris]

Stray thoughts about Girard: Mimesis, Being, Parsons, Scapegoating +

Being

Girard’s interest in mimesis goes beyond the obvious fact of imitation. It’s not simply that we imitate others, but that mimesis is about Being, a desire for Being. What is that, being? A tricky word, concept. Consider that a number of people admire some particular (potential) object of mimetic desire. Could it be that, though imitation of some object, we also desire the attention that others give to that object?

Note that this is different from conceiving a romantic attraction to Mary because you see that John is attracted to Mary. In this case you are imitating John. Whose Being are you chasing now?

Scapegoating and Parsons

From my post, 3 Manifestations of America’s vulture of violence, a speculation:

... early in my undergraduate career I read an essay that Talcott Parsons published in 1947, “Certain Primary Sources of Aggression in the Social Structure of the Western World” (reprinted in Essays in Sociological Theory), which has influenced me a great deal. Parsons argued that Western child-rearing practices generate a great deal of insecurity and anxiety at the core of personality structure. This creates an adult who has a great deal of trouble dealing with aggression and is prone to scapegoating. Inevitably, there are lots of aggressive impulses which cannot be followed out. They must be repressed. Ethnic scapegoating is one way to relieve the pressure of this repressed aggression. That, Parsons argued, is why the Western world is flush with nationalistic and ethnic antipathy.

There’s that word, “scapegoating.” But this is the first time I’ve connected Girard’s concept with Parsons. Does it work?

They are very different thinkers. I have no idea whether or not Girard was aware of Parsons, or of that particular essay. Parsons calls on Freud to make his argument. Girard was certainly aware of Freud (and, I believe, argued against him), but that’s no reason to think he knew of Parsons.

Russ Roberts: EconTalk

René Girard, Mimesis, and Conflict (with Cynthia Haven) 6/24/24

Shakespeare on the wane? What does that portend?

Drew Lichtenberg, Who’s Afraid of William Shakespeare? NYTimes, Oct. 21, 2024.

How real is this Shakespeare shrinkage? American Theatre magazine, which collects data from more than 500 theaters, publishes a list of the most performed plays each season. In 2023-24, there were 40 productions of Shakespeare’s plays. There were 52 in 2022-23 and 96 in 2018-19. Over the past five years, Shakespeare’s presence on American stages has fallen a staggering 58 percent. At many formerly Shakespeare-only theaters, the production of the Bard’s plays has dropped to as low as less than 20 percent of the repertory.

Why might American theaters be running away from Shakespeare? [...]

Over the past 10 years, as American politics and culture have grown more contentious, Shakespeare has become increasingly politicized. In 2017, the Public Theater’s Delacorte production of “Julius Caesar” depicted the assassination of a Donald Trump-like Caesar. The production elicited protests from Trump supporters, and corporate sponsors pulled their funding. Shakespeare is also under assault from the progressive left. In July 2020, the theater activist collective “We See You, White American Theater” turned the industry upside down with demands for a “bare minimum of 50 percent BIPOC representation in programming and personnel,” referring to Black, Indigenous and people of color. Though Shakespeare’s name went unmentioned, his work remained the white, male, European elephant in the room. [...]

Given contemporary political divisions, when issues such as a woman’s right to control her own body, the legacy of colonialism and anti-Black racism dominate headlines, theater producers may well be repeating historical patterns. There have been notably few productions in recent years of plays such as, “The Taming of the Shrew,” “The Tempest” or “Othello.” They may well hit too close to home.

Hmmmm.... Shakespeare has long served as something of a focal point or Schelling point in the (Western) literary system, a common point of reference. Is his work being displaced from that role? Note that Lichtenberg points out, “There is a long history of theaters running from Shakespeare during times of political division or uncertainty.” Still, where does this process go? What if there is no restoration to the center? Will Shakespeare be replaced? By whom? Or is the system transforming or even dissolving? 

I note that Lichtenberg's article seems to be about professional performances in America. What about performances in secondary schools, colleges and universities? What about performances in England, all of Britain, Western Europe, the world?

Out of my window

What hath woke wrought?

Tyler Cowen interviews Musa al-Gharbi, author of We Have Never Been Woke: The Cultural Contradictions of a New Elite.

AL-GHARBI: I have published an essay where I argue that the Great Awokening does seem to be winding down.

In the book, looking at a lot of different empirical measures, and in some of my other published research before the book, I argue that it seems like starting after 2011 with race, gender, and sexuality and stuff — but starting a year before that with Occupy Wall Street and things like this — but basically, starting after around 2010, there was this significant shift among knowledge economy professionals in how we talk and think about social justice issues. That does seem to have peaked around 2021.

Looking at the measures that I was looking at in the book, it seems like a lot of those are on the decline now, yes.

COWEN: Do we have a single coherent theory that explains both the rise of the Great Awokening and its apparent fragility? I can see that it’s easy to explain either of those, but how do we do both?

AL-GHARBI: One of the things that I argue in the book, that I think is really important for contextualizing the current moment, is that this current period of rapid change in how knowledge economy professionals talk and think about social justice and the ways we engage in politics and all of this — this moment is actually a case of something. As I show in the book, looking at the same kinds of empirical measurements, we can see that actually there were three previous episodes of great awokenings. By comparing and contrasting these cases, we can get insight into questions like, Why did they come about? Why do they end? Do they influence? Do they change anything long-term? and so on.

To that question — why did they come about, why did they end — what I argue in the book is, there seems to be two elements that are important predictors for when an awokening might come about. One of them is that they tend to happen during moments of elite overproduction, when it becomes particularly acute. This is a term drawn from Jack Goldsmith and Peter Turchin, for people who are not already familiar with it, which is basically when society starts producing more people who think that they should be elites than we have capacity to actually give those people the lives they feel like they deserve.

We have growing numbers of people who did everything right: They did all the extracurriculars, they got good grades in school, they graduated from college — even from the right college in the right majors — but they’re having a hard time getting the kinds of six-figure jobs they expected. They can’t buy a house. They’re not being able to get married and live the kind of standard of living their parents had and so on.

When you have growing numbers of elites and elite aspirants that find themselves in that position, then what they tend to do is grow really dissatisfied.

There's much more at the link.

Nicholas Confessore, The University of Michigan Doubled Down on D.E.I. What Went Wrong? NYTimes, Oct. 16, 2024. The article opens:

Leaders of the University of Michigan, one of America’s most prestigious public universities, like to say that their commitment to diversity, equity and inclusion is inseparable from the pursuit of academic excellence. Most students must take at least one class addressing “racial and ethnic intolerance and resulting inequality.” Doctoral students in educational studies must take an “equity lab” and a racial-justice seminar. Computer-science students are quizzed on microaggressions.

Programs across the university are couched in the distinctive jargon that, to D.E.I.’s practitioners, reflects proven practices for making classrooms more inclusive, and to its critics reveals how deeply D.E.I. is encoded with left-wing ideologies. Michigan’s largest division trains professors in “antiracist pedagogy” and dispenses handouts on “Identifying and Addressing Characteristics of White Supremacy Culture,” like “worship of the written word.” The engineering school promises a “pervasive education around issues of race, ethnicity, unconscious bias and inclusion.”

At the art museum, captions for an exhibit of American and European art attest to histories of oppression “even in works that may not appear to have any direct relation to these histories.” The English department has adopted a 245-word land acknowledgment, describing its core subject as “a language brought by colonizers to North America.” Even Michigan’s business school, according to its D.E.I. web page, is committed to fighting “all forms of oppression.”

A decade ago, Michigan’s leaders set in motion an ambitious new D.E.I. plan, aiming “to enact far-reaching foundational change at every level, in every unit.” Striving to touch “every individual on campus,” as the school puts it, Michigan has poured roughly a quarter of a billion dollars into D.E.I. since 2016, according to an internal presentation I obtained. A 2021 report from the conservative Heritage Foundation examining the growth of D.E.I. programs across higher education — the only such study that currently exists — found Michigan to have by far the largest D.E.I. bureaucracy of any large public university. Tens of thousands of undergraduates have completed bias training. Thousands of instructors have been trained in inclusive teaching.

And yet:

D.E.I. programs have grown, in part, to fulfill the increasingly grand institutional promises behind them: to not only enroll diverse students but also to push them to engage with one another’s differences; to not merely educate students but also repair the world outside. Under the banner of D.E.I., universities like Michigan have pledged to tackle society-wide problems: The vast disparities in private wealth, the unequal distribution of public services, the poor quality of many urban schools.

In practice, though, such ambitions can exceed the reach of even a wealthy university. Most people I spoke to at Michigan, including people who criticized other aspects of D.E.I. there, praised Wolverine Pathways, the school’s premier pipeline for underserved Michigan public-school students. Yet this year, after substantial growth, Pathways supplied just 480 undergraduates out of the 34,000 on campus. In explaining why it was so challenging to boost Black enrollment, Chavous and other school officials argued that rapidly declining high school enrollment in Detroit — a trend that was itself the product of social and economic forces beyond the university’s control — had drained Michigan’s traditional pool of Black applicants even as the school’s overall enrollment was rising.

D.E.I. theory and debates over nomenclature sometimes obscured real-world barriers to inclusion. The strategic plan for Michigan’s renowned arboretum and botanical gardens calls for employees to rethink the use of Latin and English plant names, which “actively erased” other “ways of knowing,” and adopt “a ‘polycentric’ paradigm, decentering singular ways of knowing and cocreating meaning through a variety of epistemic frames, including dominant scientific and horticultural modalities, Two-Eyed Seeing, Kinomaage and other cocreated power realignments.”

Only one sentence in the 37-page plan is devoted to the biggest impediment to making the gardens accessible to a more diverse array of visitors: It is hard to get there without a car. (While the arboretum is adjacent to campus, the gardens are some miles away.) “The No. 1 issue across the board was always transportation,” said Bob Grese, who led the arboretum and gardens until 2020. “We were never able to get funding for that.” [...]

But even some liberal scholars believe D.E.I. looms too large. Amna Khalid, a historian at Carleton College in Minnesota, argues that modern D.E.I. is not, as some on the right hold, a triumph of critical theory or postcolonialism but of the corporatization of higher education, in which universities have tried to turn moral and political ideals into a system of formulas and dashboards. “They want a managerial approach to difference,” Khalid said. “They want no friction. But diversity inherently means friction.”

There's much more at the link.

Domesticated stars, with flag

Have political parties become quasi-religious organizations?

While I don't read Brooks regularly, I do read him. Here's a passage from a recent op-ed, Why the Heck Isn’t She Running Away With This? NYTimes (Oct. 17. 2024):

Trump has spent the past nine years not even trying to expand his base but just playing to the same MAGA grievances over and over again. Kamala Harris refuses to break with Biden on any significant issue and is running as a paint-by-numbers orthodox Democrat. Neither party tolerates much ideological diversity. Neither party has a plausible strategy to build a durable majority coalition. Why?

I think the reason for all this is that political parties no longer serve the function they used to. In days gone by, parties were political organizations designed to win elections and gain power. Party leaders would expand their coalitions toward that end. Today, on the other hand, in an increasingly secular age, political parties are better seen as religious organizations that exist to provide believers with meaning, membership and moral sanctification. If that’s your purpose, of course you have to stick to the existing gospel. You have to focus your attention on affirming the creed of the current true believers. You get so buried within the walls of your own catechism, you can’t even imagine what it would be like to think outside it.

When parties were primarily political organizations, they were led by elected officials and party bosses. Now that parties are more like quasi-religions, power lies with priesthood — the dispersed array of media figures, podcast hosts and activists who run the conversation, define party orthodoxy and determine the boundaries of acceptable belief.

Many important tasks solved by transformers can't be done in subquadratic time

New paper proving that many important tasks solved by transformers can't be done in subquadratic time.

Sorry Linear Transformers, Mamba and all the others.

https://t.co/OGUEASmL23
By @firebat03 and @HantaoYu_Theory. pic.twitter.com/XlEmmtDn58

— Thomas Ahle (@thomasahle) October 14, 2024

Mechazilla chopstick catch (SpaceX)

Watch it again.

On the 5th integrated flight test of Starship the Super Heavy tower catch (better know as Mechazilla chopstick catch) SUCCEEDED AT THE FIRST TEST.

Congratulations @elonmusk and all the @SpaceX team! pic.twitter.com/Th2zzSDyv5

— Massimo (@Rainmaker1973) October 13, 2024

Once again, LLMs are shown to have difficulty with formal reasoning

1/ Can Large Language Models (LLMs) truly reason? Or are they just sophisticated pattern matchers? In our latest preprint, we explore this key question through a large-scale study of both open-source like Llama, Phi, Gemma, and Mistral and leading closed models, including the… pic.twitter.com/yli5q3fKIT

— Mehrdad Farajtabar (@MFarajtabar) October 10, 2024

The thread continues on for a total of 13 tweets. Gary Marcus discusses this study: LLMs don’t do formal reasoning - and that is a HUGE problem.

Here Come the Sun

How to Say “No” to Our A.I. Overlords

The NYTimes has a useful article about how to avoid the AI that Google, Microsoft and Meta are foisting on us:

Big tech brands like Google, Apple, Microsoft and Meta have all unleashed tech that they describe as artificial intelligence. Soon, the companies say, we’ll all be using A.I. to write emails, generate images and summarize articles.

But who asked for any of this in the first place?

Judging from the feedback I get from readers of this column, lots of people outside the tech industry remain uninterested in A.I. — and are increasingly frustrated with how difficult it has become to ignore. The companies rely on user activity to train and improve their A.I. systems, so they are testing this tech inside products we use every day.

I'm particularly worried about having these things in word-processing and email programs. I think it's particularly important that AI-generated text be clearly marked as such. This is going to make that impossible.

Nobel House [Media Notes 139]

I’m talking about the miniseries from 1988, not the James Clavell novel on which it is based. Yikes! Not sure why I sat through all four episodes, but it did.

It stars Pierce Brosnan as Ian Dunross, Tai-pan (head) of Struan & Company, a Hong Kong trading company with roots dating back to whenever. Struan’s is stretched thin financially and vulnerable for take-over. So an American corporate raider comes along and teams up with a Hong Kong rival. The game is joined by an opium dealer hoping to cash in on a compact made when Struan’s was created way back when, and the China Great Wall International Trust Corporation (in the PRC). Alongside this we have a horse race, a Chinese mole in the Hong Kong police department, a mid-level Hong Kong bank, a disloyal son, and of course, pretty women, three of them.

It's all very complicated. Intrigue abounds. Some bed-hopping. Frequent reference to “face,” “joss,” and “the way things are done in Hong Kong” (which seems to be deeply mysterious). Halfway through there’s a big fire in floating restaurant which provides a lot of spectacle, but doesn’t seem to have much effect on the plot, though many of the principle were in the restaurant at the time. But then, toward the end an apartment building collapses, killing two of the more or less bad guys, making it much easier to resolve things in favor Struan & Company. Yipee!

The whole thing was slathered in corporate intrigue and hijinks, with a couple of dashes of Orientalism and a bit of sexism on the side. Quite a piece of work.

Hossenfelder: The 2024 Nobel Prize in Physics Did Not Go To Physics -- This Physicist is very surprised

Hossenfelder comments: "A quick comment on the 2024 Nobel Prize in physics which was awarded for the basis of neural networks and artificial intelligence. Well deserved, but is it physics?" She also wonders if this does't reflect (what she regards as) the dismal state of current work in the foundations of physics and points out that physicists have been using neural networks for years as tools.

Herbert Simon won the Nobel Prize in Economics in 1978 for "for his pioneering research into the decision-making process within economic organizations." The Nobel Committee did not mention his work in Artificial Intelligence. But would he have gotten the prize without that work? 

* * * * *

Gary Marcus has an interesting post on the physics Nobel, for AI, and the chemistry as well: Two Nobel Prizes for AI, and Two Paths Forward:

Let’s start with Hinton’s award, which has led a bunch of people to scratch their head. He has absolutely been a leading figure in the machine learning field for decades, original, and, to his credit, persistent even when his line of research was out of favor. Nobody could doubt that he has made major contributions. But the citation seems to indicate that he won it for inventing back-propagation, but, well, he didn’t.

He goes on spell out a more detailed history of early work in neural nets, citing remarks by Steven Grossberg and Jürgen Schmidhuber.

Do LLMs memorize? 25% of "memorized" tokens are actually predicted using general language modeling features

Turns out LLMs don't simply memorize verbatim; 25% of "memorized" tokens are actually predicted using general language modeling features

YouTube: https://t.co/A7eMAxFLVA@arxiv: https://t.co/9KQ1Gi0VML@Bytez: https://t.co/PdSVPTz7RB@askalphaxiv: https://t.co/UK43dX1DeT pic.twitter.com/OXhW8KU6fM

— Tunadorable (@tunadorable) September 26, 2024

Abstract of the linked article:

Large Language Models (LLMs) frequently memorize long sequences verbatim, often with serious legal and privacy implications. Much prior work has studied such verbatim memorization using observational data. To complement such work, we develop a framework to study verbatim memorization in a controlled setting by continuing pre-training from Pythia checkpoints with injected sequences. We find that (1) non-trivial amounts of repetition are necessary for verbatim memorization to happen; (2) later (and presumably better) checkpoints are more likely to verbatim memorize sequences, even for out-of-distribution sequences; (3) the generation of memorized sequences is triggered by distributed model states that encode high-level features and makes important use of general language modeling capabilities. Guided by these insights, we develop stress tests to evaluate unlearning methods and find they often fail to remove the verbatim memorized information, while also degrading the LM. Overall, these findings challenge the hypothesis that verbatim memorization stems from specific model weights or mechanisms. Rather, verbatim memorization is intertwined with the LM's general capabilities and thus will be very difficult to isolate and suppress without degrading model quality.

I have a paper that deals with so-called memorization: Discursive Competence in ChatGPT, Part 2: Memory for Texts, Version 3.

Music and the Origins of Language: Neil deGrasse Tyson talks with Daniel Levitin

From the YouTube page:

Did early humans sing before they could talk? Neil deGrasse Tyson and Chuck Nice discover how music helps us recall memories, the singing Neanderthal theory. the default mode network, and how music can be used as medicine with neuroscientist and bestselling author, Daniel Levitin.

Would we have been able to communicate with aliens using music like in Close Encounters of a Third Kind? We explore Levitin’s new book I Heard There Was A Secret Chord which explores how music not only enriches our lives but also impacts our brains, behavior, and health.

We discuss how music can be a source of pleasure and how it captivates us—ever wonder why certain songs get stuck in your head? We explore how music has been a critical form of communication for thousands of years, predating written language, and how it helps encode knowledge and transmit information across generations. From ancient bone flutes to modern-day symphonies, why does music hold such a powerful place in human history?

We also dig into music's therapeutic powers—how it can boost cognitive reserves, help Parkinson's patients walk, relieve pain, and even enhance memory. Did you know that music has the power to activate every part of your brain? Whether you're soothing a baby with a lullaby or summoning old memories through a favorite song, the impact of music is profound. Levitin explains how music therapy is being explored as a potential solution to alleviate neurological afflictions like multiple sclerosis and Tourette syndrome.

Learn about the relationship between music and the brain’s "default mode network"—the state your brain enters when it’s at rest or wandering. We explore memory retrieval and how it’s tied to music’s ability to trigger unique, specific memories.

Discover why certain songs can transport us back to vivid moments in our past, acting as powerful cues for recalling experiences. We discuss how music persists beyond memory-related conditions like Alzheimer's, as seen in the case of Tony Bennett, who, despite the progression of the disease, retained the ability to perform his beloved songs. This connection between music, memory, and neural activation offers exciting possibilities for therapeutic applications in the future.

Timestamps:

00:00 - Introduction: Daniel Levitin
2:55 - Communicating to Aliens Using Music
6:12 - The Evolution of Music & Singing Neanderthal Theory
11:55 - Music v. Communication
15:45 - Neuroscience of Music & Memory Retrieval
24:34 - The Default Mode Network
28:24 - Music as Medicine
42:13 - How Does Memory Work?

How might LLMs store facts? [Multilayer Perceptrons, MLP]

Time stamps:

0:00 - Where facts in LLMs live
2:15 - Quick refresher on transformers
4:39 - Assumptions for our toy example
6:07 - Inside a multilayer perceptron
15:38 - Counting parameters
17:04 - Superposition
21:37 - Up next

Preceding videos in this series:

• But what is a GPT? Visual intro to Transformers
• Visualizing Attention: A Transformer's Heart

On the dockworkers strike, labor on the rise

Sohrab Ahmari, In Praise of the Dockworkers Shutting Down Our Ports, The Free Press, October 2, 2024.

The International Longshoremen’s Association, whose strike is crippling U.S. ports from the Gulf Coast to New England, may not seem like the wretched of the Earth. They’re asking for a 77 percent pay increase on top of the $39 per hour those on the top tiers already make. The union’s president, Harold Daggett, earns $728,000 a year and once owned a 76-foot boat. With major disruptions looming, no wonder even some of those Americans ordinarily sympathetic to organized labor might be thinking, Okay, this is going too far. The less sympathetic are already calling for the Marines to suppress the strike.

But here’s the hard truth: The militancy showcased by the ILA is exactly what is needed to restore a fairer, more balanced economy—the kind that created the middle class in the postwar decades and allowed your grandparents to access reliable healthcare, take vacations, and enjoy disposable incomes. Those who complain that today’s left has come to privilege boutique identity politics over bread-and-butter concerns should cheer the longshoremen. There is nothing “woke” about their exercise of economic power to win material gains for themselves and their industrial brethren.

The longshoremen are striking for familiar reasons: better wages and benefits, and to prevent automation from decimating their livelihoods. [...]

Some critics argue that the ILA’s demand that no automation take place at the ports is unreasonably rigid. It’s certainly audacious, but it’s called an opening gambit for a reason. I suspect we will see concessions on both sides leading to a reasonable settlement, as in the case of SAG. The rest—gripes about how much the ILA president earns or how longshoremen are already well-compensated—is the tired propaganda of the C-suite class. [...]

The ILA strike is a rare reminder of working people’s power to shut it all down. [...] Real progress in market societies results from precisely this dynamic tension between labor and capital. For too long, however, one side of the equation—labor—has been torpid, not to say dormant. The asset-rich had it so good over the past few decades—capturing the lion’s share of the upside from de-unionization, financialization, and offshoring, as wages stagnated for the bottom half—that they all but forgot what labor militancy can look and sound like. How much it can sting.

Now, the labor movement is on the move. Since the pandemic, workers across a wide range of industries have joined arms to form new unions or to secure better wages and working conditions under existing collective-bargaining agreements. Last year, some 539,000 workers were involved in 470 strikes and walkouts, according to Cornell researchers, up from 140,000 workers mounting 279 strikes in 2021. This ferment—what one labor scholar has called a “strike wave”—comes after the union share of the private-economy workforce has declined from its peak of one-third in 1945 to 6 percent today.

There’s more at the link.

Blown-out flicks of flowers

Problems with so-called AI scaling laws

Arvind Narayanan and Sayash Kapoor, AI Scaling Myths, AI Snake Oil, June 27, 2024. The introduction:

So far, bigger and bigger language models have proven more and more capable. But does the past predict the future?

One popular view is that we should expect the trends that have held so far to continue for many more orders of magnitude, and that it will potentially get us to artificial general intelligence, or AGI.

This view rests on a series of myths and misconceptions. The seeming predictability of scaling is a misunderstanding of what research has shown. Besides, there are signs that LLM developers are already at the limit of high-quality training data. And the industry is seeing strong downward pressure on model size. While we can't predict exactly how far AI will advance through scaling, we think there’s virtually no chance that scaling alone will lead to AGI.

Under the heading, "Scaling “laws” are often misunderstood", they note:

Scaling laws only quantify the decrease in perplexity, that is, improvement in how well models can predict the next word in a sequence. Of course, perplexity is more or less irrelevant to end users — what matters is “emergent abilities”, that is, models’ tendency to acquire new capabilities as size increases.

Emergence is not governed by any law-like behavior. It is true that so far, increases in scale have brought new capabilities. But there is no empirical regularity that gives us confidence that this will continue indefinitely.

Why might emergence not continue indefinitely? This gets at one of the core debates about LLM capabilities — are they capable of extrapolation or do they only learn tasks represented in the training data? The evidence is incomplete and there is a wide range of reasonable ways to interpret it. But we lean toward the skeptical view.

There is much more under the following headings:

• Trend extrapolation is baseless speculation
• Synthetic data is not magic
• Models have been getting smaller but are being trained for longer
• The ladder of generality

These remarks are from the section on models getting smaller:

In other words, there are many applications that are possible to build with current LLM capabilities but aren’t being built or adopted due to cost, among other reasons. This is especially true for “agentic” workflows which might invoke LLMs tens or hundreds of times to complete a task, such as code generation.

In the past year, much of the development effort has gone into producing smaller models at a given capability level. Frontier model developers no longer reveal model sizes, so we can’t be sure of this, but we can make educated guesses by using API pricing as a rough proxy for size. GPT-4o costs only 25% as much as GPT-4 does, while being similar or better in capabilities. We see the same pattern with Anthropic and Google. Claude 3 Opus is the most expensive (and presumably biggest) model in the Claude family, but the more recent Claude 3.5 Sonnet is both 5x cheaper and more capable. Similarly, Gemini 1.5 Pro is both cheaper and more capable than Gemini 1.0 Ultra. So with all three developers, the biggest model isn’t the most capable!

Training compute, on the other hand, will probably continue to scale for the time being. Paradoxically, smaller models require more training to reach the same level of performance. So the downward pressure on model size is putting upward pressure on training compute.

Check out the newsletter, AI Snake Oil, and the book of the same title.

OpenAI’s $6.6B raise: What were they thinking?

Cory Weinberg, The Briefing: The Cynic’s Guide to OpenAI’s Megaround, The Information, Oct. 2, 2024:

The biggest question is: Will OpenAI ever be a good business? It’s debatable right now. At least on a sales multiple basis (13 to 14 times next year’s forecasted $11.6 billion revenue), some investors can justify it without embarrassment.

But investors in the latest round probably need OpenAI to eventually become a roughly $1 trillion company to get a strong return. That means at some point the startup will have to become a cash flow machine rather than a cash incinerator.

Of the seven companies with over $1 trillion in market cap currently, the median free cash flow from the past year was $57 billion. In that regard, OpenAI, which is chasing growth and spending heavily on computing capacity, has quite a way to go. (For what it’s worth, Fidelity investing in the latest round should mean we get a regular check-in on how OpenAI’s valuation is shifting, at least in the opinion of Fidelity, which needs to make its startup valuations public.)

To be sure, even if OpenAI’s latest benefactors don’t believe it can get to $1 trillion, many of them have all sorts of ulterior, strategic reasons to back the startup.