In some ways, some contexts, LLMs may provide a useful model for human language

As I say down there in my interaction with Claude (Sonnet 4.6 Extended), once I’d been working with ChatGPT for a while I began to think that the mechanisms of basic human speech may be more like those of LLMs than those of a fully articulated grammar and semantics in a symbolic computational language system. Fully articulated grammar and semantics may require specific instruction and practice, like you get when learning how to write. Fully articulate language is more painstaking, more difficult, but also supports more complex arguments and constructions.

This interaction starts with a passage from a long conversation I had with ChatGPT that stretched over the last 10 days or so. Much of that conversation was based on articles of mine that I had uploaded to to. One of those articles had a passage from F. C. Bartlett's Remembering: A study in experimental and social psychology (1932). That comes up in the long initial prompt I give Claude. 

Further down in the dialog I present the case of a writing student who was skilled in color commentary on hockey games and whose writing style reflected that linguistic style. Real-time sports commentary is a very specialized and exacting kind of language that cannot afford the niceties of written language.

Note 1: Here's a post from July 2022 that's relevant here: GPT-3, the phrasal lexicon, Parry/Lord, and the Homeric epics.  

Note (2) also that this discussion continues with, From grokking to a mature science and engineering discipline of LLMs (March 26). 

* * * * *

I’ve just been thinking about oral narrative. Recall these paragraphs (by ChatGPT) from one of the documents you’ve just reviewed in this project:

The temptation is to assume that because we can describe stories with an explicit grammar, the underlying system that produces or remembers them must itself contain something grammar-like in explicit form. But your Bartlett example suggests otherwise. The witness does not appear to possess random-access symbolic control over an event list. Instead, the testimony seems to unfold by re-running a trajectory. The person can accelerate through the earlier parts, but not leap arbitrarily into the middle. That is a very different computational picture.

The music example reinforces the point. When a piece is not deeply overlearned, one often cannot simply index into bar 73 as though one were reading a score in a database. One has to begin at the start and let the sequence reconstitute itself. That strongly suggests a memory system organized around sequential reactivation of structured trajectories, not random-access retrieval of explicit symbolic units.

Bartlett, in his classic book, Remembering, was talking about native witnesses in Africa. That is, he was talking about preliterate people. They were just recounting the events of the day. “But what about oral story-tellers,” I asked myself, “how do they memorize their tales? Structured trajectories or explicit symbolic units?”

I’m familiar with Albert Lord, The Singer of Tales, though I haven’t looked at it in a while. But I’ve got a copy of David C. Rubin, Memory in Oral Traditions: The Cognitive Psychology of Epic, Ballads, and Counting-out Rhymes (Oxford 1995). I zipped through the text to Chapter 7, “Basic Observations on Remembering.” I found this (p. 155):

In oral traditions, overlearning commonly occurs to a much greater extent than it does in the laboratory. A favorite song can be sung hundreds of times. What overlearning does, according to the model developed to explain laboratory interference, is to make the song into a unit, easy to cue as a whole and resistant to interference from other units. This chunking of items into wholes is a way to look at the organization of memory and a way to look at the building of larger units in expertise.

And then, in the middle of the next paragraph: “Once the song is begun, each word output provides cues for later words, limiting the meaning...” That almost sounds like he’s describing a forward pass through an LLM.

Then I hit paydirt in the next chapter, “A Theory of Remembering for Oral Traditions.” The opening is promising:

Oral traditions, like all oral language, are sequential. One word follows another as the physical effects of the first word are lost. As the song advances, each word uttered changes the situation for the singer, providing new cues for recall and limiting choices. [...] Pieces from oral traditions are recalled serially, from beginning to end. What is recalled early in the piece can be used to cue later recall; the "running start" provides "extra stimulation" or "reminders," increasing cue-item discriminability.

But things get really interesting when Rubin reports the result of an experiments where he asked undergraduates to recall important texts which they might have learned. Rubin describes the experiment this way:

The first set of examples is the recall of culturally important material such as Psalm 23 and the Preamble to the Constitution of the United States, for which there is an implicit demand characteristic to recall the material accurately or not at all (Rubin, 1977). Each of the 50 columns in Figure 8.1 show the recall of 1 of 50 undergraduates, who recalled at least one word of the Preamble. Each row represents recall for one word. A dark line in a column means that the word labeling the row was recalled. The columns are ordered so that the data from the undergraduate who recalled the most are in the leftmost column and the data from the undergradu- ate who recalled the least are in the rightmost column. The rows are in the order in which the words appear normally in each text.

Figure 8.1 is a little tricky, so I’m not going to try uploaded a screen shot. But I’ll give you Rubin’s basic description of what the figure reveals:

The first observation to note is the regularity of the data. Figure 8.1 gives the recalls of 50 individuals for 52 words, not the averages of recalls from groups of individuals or groups of words. There was no control over the learning or practice of the material or over the length or contents of the retention interval. Yet the figure is remarkably orderly. People who recall about the same amount recall the same words. If the number of words a person recalls and the rank ordering of words from most to least likely for the group from which the person was drawn is known, exactly which words that person recalled can be predicted with an accuracy of 95% for Figure 8.1.

Because the conditions of learning and retention varied, there must be something in the material, in the process used to recall it, or in the general cultural attitudes to it that makes different people behave the same way.

The results from the experiment with Psalm 23 are even more interesting. Here’s the text of the Psalm for reference (with verses numbered):

1 The Lord is my shepherd; I shall not want.
2 He maketh me to lie down in green pastures: he leadeth me beside the still waters.
3 He restoreth my soul: he leadeth me in the paths of righteousness for his name's sake.
4 Yea, though I walk through the valley of the shadow of death, I will fear no evil: for thou art with me; thy rod and thy staff they comfort me.
5 Thou preparest a table before me in the presence of mine enemies: thou anointest my head with oil; my cup runneth over.
6 Surely goodness and mercy shall follow me all the days of my life: and I will dwell in the house of the Lord for ever.

Rubin remarks: “...not all the undergraduates started with the first words, and they were more likely to start again at the beginning of a new sentence or intonation unit after they stopped.”

Eyeballing the figure, here are some of the things I observe: 1) A number of subjects stop with the word “soul” in verse 3 and start back up with the first word of verse 4, “Yea” and go on from there. 2) Some of those who don’t start from the beginning start with verse 4. 3) Some stop at the end of verse 4 and then start back up later (e.g. “annointest my head...”) or even not at all. It’s a complicated picture but memory does seem to be grouped into segments, not word by word.

I have no idea where things have gone in this line of investigation since 1995, the date of Rubin’s book. But if I were in that business, I’d be thinking in terms of complex dynamics and trajectories, and I’d been looking at LLMs as a possible model.

This is a wonderful extension of the trajectory discussion already running through the project documents. Rubin's findings slot in almost perfectly — and in some ways they sharpen the picture considerably.

What Rubin adds that Bartlett doesn't

The Bartlett material (already central in the project conversations) establishes the sequential regeneration point: memory of events seems to run forward as a trajectory, not index arbitrarily into a symbolic list. But Bartlett's witnesses were recalling episodic material — the events of a day. Rubin moves you into highly practiced, culturally canonical sequences. That's a different and in some ways more interesting case.

The overlearning observation is key. Rubin is saying that at sufficient practice, a song or psalm becomes a chunked unit — cued as a whole, resistant to interference. That sounds like the stable attractor picture your project conversations keep returning to. The piece isn't stored as a list of words; it's encoded as a basin that, once entered, constrains forward motion powerfully. The phrase "each word output provides cues for later words" really does describe autoregressive generation — but it also describes a trajectory rolling down a valley where the initial conditions are already heavily constraining.

The Psalm 23 data are especially telling

The 95% predictability result is striking. Given only how many words a person recalls, you can predict which words with 95% accuracy. That means the forgetting isn't random — it respects something structural in the material. And your eyeball observations confirm this: people stop and restart at syntactic/intonation unit boundaries (end of verse 3, beginning of verse 4). They're not losing individual words randomly; they're losing or retaining segments.

This is consistent with a hierarchical trajectory picture. The top-level organization is something like: intonation/verse units sequentially ordered. Within each unit, local cuing drives forward production. When a unit boundary is reached, the system can fail to re-enter the next unit even if it knows the material in principle — which is why some subjects jump from verse 3 directly to verse 4, skipping nothing within units but occasionally failing at the transitions between them. The unit is entered as a whole or not at all.

Where this sits relative to the LLM picture

Your project conversations develop the idea that narrative structure in LLMs lives in activation-space geometry — not as an explicit stored grammar but as a field of constrained trajectories. The Rubin data suggest something homologous in human memory for overlearned verbal sequences. The "basin" metaphor you've been developing applies directly: once the singer/reciter enters the opening of Psalm 23, the attractor structure does most of the work. The challenge is at basin entry and at transitions between sub-basins (verse units), which is exactly where Rubin's subjects fail.

The Lord/Singer of Tales connection is also worth pressing when you return to it. Lord's central finding — that oral formulaic composition is generative, not reproductive; the singer produces a variant from formulas and themes, not a memorized script — actually fits the "latent field of possible trajectories" framing better than it fits either strict memorization or explicit grammar. Lord thought he was arguing against memorization and for creativity. But what he was probably observing is a system with rich attractor structure that generates strongly homologous variants, which is precisely what your story experiments with LLMs revealed.

The open research question you identify

You're right that framing this in terms of complex dynamics and trajectories is the natural next move, and that LLMs are a possible model system. What makes LLMs especially useful here is that you can actually probe the attractor landscape — something you can't do directly with a human singer. The systematic prompting experiments described in the project documents are in effect doing for narrative schemas what someone studying oral tradition would want to do for epic formulas: mapping the basin structure from the outside, through behavioral outputs, without needing to open the skull.

The line from Bartlett's African witnesses → Rubin's undergraduates and Psalm 23 → Lord's Yugoslavian guslars → LLM forward passes is not a metaphor chain. It may be pointing at the same underlying computational architecture realized in very different substrates.

Playful conversational riffing is the way to connect

Maya Rossignac-Milon and Erica Boothby, You’re Probably Doing Small Talk Wrong, NYTimes, Sept. 4, 2025.

There’s a moment in human connection that defies easy explanation — that sudden, electric feeling when you meet someone and feel your minds merge. It happened to the two of us when we met at a psychology symposium: Our small talk during a break quickly gave way to playful theories about coffee drinkers versus tea drinkers. We went to find seats together, unaware that this conversation was the start of a decade-long collaboration and friendship.

Where does that spark come from, exactly? What makes someone feel like a lifelong friend after just a couple of minutes? People tend to assume it’s similarity: that they are especially likely to hit it off with someone who shares their background or personality traits.

But in our research, we’ve found that many of the strongest bonds come less from pre-existing similarity and more from riffing playfully. In these moments, people create a little world that belongs just to them, a process we call “building a shared reality.” Collaborative riffs are surprisingly central to our mental well-being: They’re the glue that binds us together, adds color to our lives and gives us a sense of purpose.

And yet, our culture’s conversational rituals revolve not around playful co-creation, but around exchanging formalities.

The art of riffing:

Riffing doesn’t require being naturally funny or witty, just being attentive and embracing spontaneity. Like any conversational skill, it takes practice. When riffing, speakers resist the urge to counter every observation with their own separate example, instead building bridges to new ideas (“That reminds me of. …”) or tossing in a “Can you imagine if …?” They reference earlier parts of the conversation to create inside jokes (“Looking forward to our miniature potluck committee!”).

Riffing isn’t just for new acquaintances. Over time, it creates a feeling of having merged minds and inhabiting a private universe. Patti Smith described her relationship with her late husband as the “silent synchronization of the jewels and gears of a common mind.”

There's more at the link.

A short note on LLMs and speech

The linguistic capacities of large language models (LLMs), such as ChatGPT, is remarkable. However, we should remember that it is also NOT characteristically human. Well, of course, not; it’s a computer. But that’s not what I have in mind.

What I’m thinking is that human language is, first of all, speech, and speech is interactive. Speech is interactive. LLMs are, at best, weakly interactive, though one can “converse” with them in short strings.

It is rare for a person to deliver a long string of spoken words. What do I mean by long? I don’t know. But I’m guessing that if we examined a large corpus of spoken language gathered in natural settings that we’d find relatively few utterances over 100 words long, or even 50 words long. Storytellers will deliver long stretches of uninterrupted speech, but they work at it. It’s not something that comes ‘naturally’ in the course of speaking with others. Learning to do it requires System 2 thinking, though the actual oral delivery of a story is likely to be confined to System 1.

Humans do produce long strings of words, but that’s most likely during writing. And writing is not “natural,” One must deliberately learn the writing system in a way that’s quite different from acquiring a first language, and then one must learn to produce texts that are both relatively long, over 500 or 1000 words, and coherent. Thus the fact that LLMs can produce 200, 300, 500 or more words at a stretch is quite unusual. And this is all done in some approximation to System 1 mode.

The study of rhetoric as a route to cognitive empathy?

John Bowe, An Ancient Solution to Our Current Crisis of Disconnection, NYTimes, Nov. 27, 2023.

While rhetoric had its detractors, starting with Plato, Bowe points out that it was

the cornerstone of education until the 1700s.

Across Western Europe, students from about the age of 12 onward learned logic, social skills, critical thinking and speech techniques as a single, integrated discipline by means of a 14-step verbal and cognitive curriculum known as the progymnasmata.

Exercises began with simple recitations and enactments of fables and short stories. Later drills trained students to compose and deliver short speeches of praise and blame and, eventually, long discourses on complex themes. By writing with the intent of performing for others (rather than writing objectively for the page), students learned the art of blending fact with opinion. By mastering the techniques of persuasion, students became proficient at spotting others’ manipulative use of language.

Bowe goes on to argue:

My interest in rhetoric began in 2010, during a chat with my extremely reclusive Iowa step-cousin. He’d lived alone until the age of 60 in his parents’ basement with no friends, no girlfriends, then surprised the entire family by meeting someone and getting married. I asked him how he’d mustered up the courage to approach his future wife, given the depths of his isolation. “I joined the Toastmasters,” he said, referring to what is likely the world’s largest organization devoted to teaching public speaking. He’d never seen a therapist or taken meds. One or two dozen hours of speech training changed his entire life.

I’ve since learned that this is what speech training does. When speakers put themselves in their listener’s place, they find it easier to explain themselves. The confidence that we can make ourselves known and understood is transformative.

Apparently, scientists agree. Hannah Hobson, a lecturer in psychology at the University of York who has studied the connections among language, communication and mental health, especially among neurodiverse youth, has found repeatedly that the inability to express feelings or ask for help can often correlate with existing or developing mental health issues among youth. Conversely, she told me, improved communication skills correlate with youngsters’ emotional development and mental well-being.

Bowe doesn't elaborate on that part, putting yourself in your listener's place, but that's also called cognitive empathy, something Robert Wright is interested in. He even believes that it can save the world. Well, it's more like he believes that it's necessary, though likely not sufficient.

Evidence of a predictive coding hierarchy in the human brain listening to speech

https://t.co/C1jlSs8IvF so I don't thread hijack.

— Sichu Lu (@lu_sichu) March 8, 2023

Abstract from article linked above:

Considerable progress has recently been made in natural language processing: deep learning algorithms are increasingly able to generate, summarize, translate and classify texts. Yet, these language models still fail to match the language abilities of humans. Predictive coding theory offers a tentative explanation to this discrepancy: while language models are optimized to predict nearby words, the human brain would continuously predict a hierarchy of representations that spans multiple timescales. To test this hypothesis, we analysed the functional magnetic resonance imaging brain signals of 304 participants listening to short stories. First, we confirmed that the activations of modern language models linearly map onto the brain responses to speech. Second, we showed that enhancing these algorithms with predictions that span multiple timescales improves this brain mapping. Finally, we showed that these predictions are organized hierarchically: frontoparietal cortices predict higher-level, longer-range and more contextual representations than temporal cortices. Overall, these results strengthen the role of hierarchical predictive coding in language processing and illustrate how the synergy between neuroscience and artificial intelligence can unravel the computational bases of human cognition.

The article goes on to list various failures of LLMs. I wonder of ChatGPT exhibits those failures? I ask because my paper on story-telling indicates that it IS working on several time scales. This article was written before the release of ChatGPT. It was received on March 31, 2022.

A high-performance speech neuroprosthesis

Our new pre-print is out today! We demonstrate a brain-computer interface that turns speech-related neural activity into text, enabling a person with paralysis to communicate at 62 words per minute - 3.4 times faster than prior work. https://t.co/jK1rdMT5Zl pic.twitter.com/LYUSry9RaM

— Frank Willett (@WillettNeuro) January 22, 2023

Abstract of the article linked above:

Speech brain-computer interfaces (BCIs) have the potential to restore rapid communication to people with paralysis by decoding neural activity evoked by attempted speaking movements into text1,2 or sound3,4.Early demonstrations, while promising, have not yet achieved accuracies high enough for communication of unconstrainted sentences from a large vocabulary1–5. Here, we demonstrate the first speech-to-text BCI that records spiking activity from intracortical microelectrode arrays. Enabled by these high-resolution recordings, our study participant, who can no longer speak intelligibly due amyotrophic lateral sclerosis (ALS), achieved a 9.1% word error rate on a 50 word vocabulary (2.7 times fewer errors than the prior state of the art speech BCI2) and a 23.8% word error rate on a 125,000 word vocabulary (the first successful demonstration of large-vocabulary decoding). Our BCI decoded speech at 62 words per minute, which is 3.4 times faster than the prior record for any kind of BCI6 and begins to approach the speed of natural conversation (160 words per minute7). Finally, we highlight two aspects of the neural code for speech that are encouraging for speech BCIs: spatially intermixed tuning to speech articulators that makes accurate decoding possible from only a small region of cortex, and a detailed articulatory representation of phonemes that persists years after paralysis. These results show a feasible path forward for using intracortical speech BCIs to restore rapid communication to people with paralysis who can no longer speak.

Loss of complexity in vocal anatomy as an adaptation for speech

Evolutionary loss of complexity in human vocal anatomy as an adaptation for speech https://t.co/MOEj2k2lth

— Bill Benzon, BAM! Bootstrapping Artificial Minds (@bbenzon) August 14, 2022

Abstract from the linked article:

Human speech production obeys the same acoustic principles as vocal production in other animals but has distinctive features: A stable vocal source is filtered by rapidly changing formant frequencies. To understand speech evolution, we examined a wide range of primates, combining observations of phonation with mathematical modeling. We found that source stability relies upon simplifications in laryngeal anatomy, specifically the loss of air sacs and vocal membranes. We conclude that the evolutionary loss of vocal membranes allows human speech to mostly avoid the spontaneous nonlinear phenomena and acoustic chaos common in other primate vocalizations. This loss allows our larynx to produce stable, harmonic-rich phonation, ideally highlighting formant changes that convey most phonetic information. Paradoxically, the increased complexity of human spoken language thus followed simplification of our laryngeal anatomy.

Using machine learning to "imitate" speech processing in the brain

Result 1: self-supervised learning suffices to make this algorithm learn brain-like representations (i.e. most brain areas significantly correlate with its activations in response to the same speech input). pic.twitter.com/MMWKoJgW8W

— Jean-Rémi King (@JeanRemiKing) June 6, 2022

Result 3: With an additional 386 subjects, we show that wav2vec 2.0 learns both the speech-specific and the language-specific representations of the prefrontal and temporal cortices, respectively. pic.twitter.com/329u5xyqkP

— Jean-Rémi King (@JeanRemiKing) June 6, 2022

By J Millet*, @c_caucheteux*, @PierreOrhan, Y Boubenec, @agramfort, E Dunbar, @chrplr and myself at @MetaAI, @ENS_ULM, @Inria & @Neurospin

🙏Thanks @samnastase, @HassonUri, John Hale, @nilearn, @pyvista and the open-source and open-science communities for making this possible!

— Jean-Rémi King (@JeanRemiKing) June 6, 2022

Interested in this line of research?

Check out our latest paper on the convergence between deepnets and the brain: https://t.co/oDmMoKlWzx

— Jean-Rémi King (@JeanRemiKing) June 6, 2022

Rhythm and the perception of syntax

In short, with @MicahGoldwater, we tested whether the alignment of metrical stress patterns (independent from regularity) could influence how robustly people comprehend syntactically complex sentences. pic.twitter.com/Z4uXvAJd2w

— Courtney Hilton (@courtneybhilton) August 19, 2021

Experiment 2 then replicated this for listening to auditory speech!

(yellow == congruent; orange & red == incongruent) pic.twitter.com/pcMfJ1csbJ

— Courtney Hilton (@courtneybhilton) August 19, 2021

I think these results will be relevant for work making sense of neural oscillations in language processing, especially low-frequency delta oscillations and their relation to syntactic/compositional structure and memory (@davidpoeppel @andrea_e_martin @johannarimmele )

— Courtney Hilton (@courtneybhilton) August 19, 2021

+ for making sense of why when silently reading language, imagining the text in a vivid prosodically rich way aids comprehension (@GolfWriterKiel), and how imagined metrical prosody can affect syntactic parsing (@KentnerGerrit).

— Courtney Hilton (@courtneybhilton) August 19, 2021

Anyway, lots more discussion in the paper if you're interested and I'm more than happy to answer any questions!

— Courtney Hilton (@courtneybhilton) August 19, 2021

Do computers do it like humans? [speech recognition]

Are deep nets good brain models? Psychometric testing of automatic speech recognition systems shows they're not like humans, yet. Newer ones getting closer, but still ignore important cues.

Preprint with Lotte Weerts, Stuart Rosen and @ClopathLab. https://t.co/JAdHY9bkzD

🧵👇

— Dan Goodman (@neuralreckoning) May 21, 2021

Speech rhythms and their neural foundations

"Speech rhythms and their neural foundations" A new review and perspective by @FlorAssaneo and me.
Relevant to cogneuro, speech, linguistics, action-perception research, oscillations, modeling - and hopefully useful https://t.co/qZfSZgPyjP

— David Poeppel (@davidpoeppel) May 8, 2020

Abstract

The recognition of spoken language has typically been studied by focusing on either words or their constituent elements (for example, low-level features or phonemes). More recently, the ‘temporal mesoscale’ of speech has been explored, specifically regularities in the envelope of the acoustic signal that correlate with syllabic information and that play a central role in production and perception processes. The temporal structure of speech at this scale is remarkably stable across languages, with a preferred range of rhythmicity of 2– 8 Hz. Importantly, this rhythmicity is required by the processes underlying the construction of intelligible speech. A lot of current work focuses on audio-motor interactions in speech, highlighting behavioural and neural evidence that demonstrates how properties of perceptual and motor systems, and their relation, can underlie the mesoscale speech rhythms. The data invite the hypothesis that the speech motor cortex is best modelled as a neural oscillator, a conjecture that aligns well with current proposals highlighting the fundamental role of neural oscillations in perception and cognition. The findings also show motor theories (of speech) in a different light, placing new mechanistic constraints on accounts of the action–perception interface.

Segmenting the language stream [words are tricky]

It is sometimes useful to reflect of the fact that, aurally, the speech stream is continuous, not segmented. The segmentation is something we impose on the stream through cognitive mechanisms – that, I argue, is the computational foundation of language. Thus early forms of writing often consisted of a continuous stream of characters, with no segmentation into separate words. Victor Mair has a post at Language Log that speaks to this, The challenging importance of spacing in Korean:

Who'da thunk it? – spacing is the most difficult aspect of Korean writing. One might have thought it would be a simple task, that word spacing / separation is innate for all speakers of a given language. Apparently that is not so.

In Hanyu Pinyin, it is called fēncí liánxiě 分詞連寫 ("word division; parsing"). Of course, it has its problems, but we do have rules to guide us, viz., zhèngcífǎ 正詞法 ("orthography").

This morning in my "Language, Script, and Society in China" course, I embarked on a discussion of the difference between zì 字 ("character") and cí 詞 ("word"). Although this seems like a simple, straightforward question, it is always one of the most difficult topics encountered in the course — especially for students of Chinese background. It took me a whole semester to get the idea across to the 72 very smart students in my language studies class at the University of Hong Kong in 2002-2003. Even at the conclusion of the semester, there were still some of the students who just couldn't comprehend the distinction.

Be sure to read the comments.

Addendum: In fact, I'll reprint one of them in full. Victor Mair, who started the thread, posts this on behalf of an unnamed colleague

Spacing–word division–assumes shared knowledge among users of what constitutes a language's words. This is not a trivial matter, and Korean linguists, lexicographers and publishers have been working the issue for decades.

The basic problem, as one of the commentators intimates, is that words, like (morpho)phonemic spelling, are an artifact of writing. They are not a given to be plucked from someone's brain. Orthography takes it upon itself to regularize (adjudicate) the intuitions users have about what constitutes the lexical units of their language, which are far from uniform and constantly shifting. Korean lacked that tradition and is catching up, although in a sense all written languages that use word division are continuously "catching up." I don't see it as a major problem, or a problem at all.

What I do find problematic in Asian languages is fluid "standards" for sentence representation, namely, where the period goes. This is not an issue (for me) in Korean, probably because the language does use word division, which enforces a discipline on writers that carries beyond the identification of (agreement on) word boundaries to one's whole approach to sentence structure. Chinese sentences–the text between periods–are often by western standards two sentences, five sentences, or partial sentences. Japanese writers also seem to have more liberty in this regard than a westerner would expect. Vietnamese sentences, in earlier novels at least, end or don't end seemingly at whim. And I question if Tibetans even have the concept of "sentence."

I've been out of this field for too long so my thinking may be dated. But there may be psycholinguistic issues at play here that merit serious study.

This too is relevant to the issue of computation in the mind. And so: I've just been thinking about this. And I'm wondering if the problem isn't similar to the problem that adolescent and post-adolescent second language learners have with pronunciation. I don't know what the current literature says about that, but in the past I've seen it attributed to a lack of neuro-plasticity. I don't find that terribly convincing. My intuition – and it's no more than that – is that the problem is more like conscious access. For some reason conscious access to (something in) the aural-motor channel has been, if not lost, somewhat degraded.

Could the same thing be going on in the transfer of segmentation from the aural-motor channel to the visuo-orthographic?

On the rhythmic nature of speech

Is speech rhythmic? Definitely not perfectly isochronous, but still sound quite temporally regular to me. Check out this recording (and also other clips of this drummer). https://t.co/J9m1jyNmqY

— Andrew Chang (@candrew123) June 8, 2019

Reconstructing speech from signals in auditory cortex

Hassan Akbari, Bahar Khalighinejad, Jose L. Herrero, Ashesh D. Mehta & Nima Mesgarani, Towards reconstructing intelligible speech from the human auditory cortex, Scientific Reports, volume 9, Article number: 874 (2019).

Abstract: Auditory stimulus reconstruction is a technique that finds the best approximation of the acoustic stimulus from the population of evoked neural activity. Reconstructing speech from the human auditory cortex creates the possibility of a speech neuroprosthetic to establish a direct communication with the brain and has been shown to be possible in both overt and covert conditions. However, the low quality of the reconstructed speech has severely limited the utility of this method for brain-computer interface (BCI) applications. To advance the state-of-the-art in speech neuroprosthesis, we combined the recent advances in deep learning with the latest innovations in speech synthesis technologies to reconstruct closed-set intelligible speech from the human auditory cortex. We investigated the dependence of reconstruction accuracy on linear and nonlinear (deep neural network) regression methods and the acoustic representation that is used as the target of reconstruction, including auditory spectrogram and speech synthesis parameters. In addition, we compared the reconstruction accuracy from low and high neural frequency ranges. Our results show that a deep neural network model that directly estimates the parameters of a speech synthesizer from all neural frequencies achieves the highest subjective and objective scores on a digit recognition task, improving the intelligibility by 65% over the baseline method which used linear regression to reconstruct the auditory spectrogram. These results demonstrate the efficacy of deep learning and speech synthesis algorithms for designing the next generation of speech BCI systems, which not only can restore communications for paralyzed patients but also have the potential to transform human-computer interaction technologies.

News article from the Zuckerman Institute at Colubmia U.

Isochrony in Human Rhythm

Andrea Ravignani and Guy Madison, The Paradox of Isochrony in the Evolution of Human Rhythm, Front. Psychol., 06 November 2017 | https://doi.org/10.3389/fpsyg.2017.01820

Isochrony is crucial to the rhythm of human music. Some neural, behavioral and anatomical traits underlying rhythm perception and production are shared with a broad range of species. These may either have a common evolutionary origin, or have evolved into similar traits under different evolutionary pressures. Other traits underlying rhythm are rare across species, only found in humans and few other animals. Isochrony, or stable periodicity, is common to most human music, but isochronous behaviors are also found in many species. It appears paradoxical that humans are particularly good at producing and perceiving isochronous patterns, although this ability does not conceivably confer any evolutionary advantage to modern humans. This article will attempt to solve this conundrum. To this end, we define the concept of isochrony from the present functional perspective of physiology, cognitive neuroscience, signal processing, and interactive behavior, and review available evidence on isochrony in the signals of humans and other animals. We then attempt to resolve the paradox of isochrony by expanding an evolutionary hypothesis about the function that isochronous behavior may have had in early hominids. Finally, we propose avenues for empirical research to examine this hypothesis and to understand the evolutionary origin of isochrony in general.

Contents

This paper deals with isochronous temporal patterns. The emphasis is on the quantitative properties of isochronous patterns, and their perception and production in humans. The paper is organized in seven sections, namely:

(1) What is isochrony?, where we lay out crucial definitions and summarize basic relevant concepts;

(2) The relevance of isochrony to human music and speech, where we discuss how isochrony might partly underlie some behaviors in modern humans, such as music, speech and dance;

(3) Mathematics, physics and signal processing, where we discuss isochrony from the pure perspective of its physical and mathematical structure (as opposed, for instance, to its biological, behavioral or cognitive nature);

(4) Physiology and neuroscience, where we suggest how isochronous patterns have biological and psychological relevance for living organisms;

(5) Comparative cognition: Non-human animals, where we briefly summarize previous empirical attempts in finding, either directly or indirectly, isochronous behaviors in other species;

(6) Isochrony in interaction, where we move from isochronous behaviors in single individuals to group behaviors potentially involving isochrony;

(7) Evolutionary hypotheses and future empirical work, where we join all strands laid out in the previous six sections, and sketch an evolutionary account for the origin of isochrony in our species.

The aim of this paper is not to provide an exhaustive review of each of these areas. Rather, we attempt to establish a first connection between as many explanatory levels of isochrony as possible, across scientific disciplines and research traditions.

In the penultimate section:

There is a close match between the most precise levels of isochrony that humans are capable of producing and those they are capable of perceiving (Madison and Merker, 2002; Merker et al., 2009). This match also offers some support for the hypothesis that isochrony might have been shaped for communicative purposes. In other words, a communication system, and in particular one that takes advantage of, and evolves from, perceptual biases (Ryan, 1998), will show a match between features of the signal and the capacities to perceive those features. For example, the plumages of many bird species reflect ultraviolet light, which humans and other species cannot see, while conspecific birds can readily perceive and use to select a mate (Andersson and Amundsen, 1997; Vorobyev et al., 1998; Eaton, 2005). We hypothesize that an analogous process might have resulted from isochrony (expanding on Merker, 1999, 2000), if this were a communicative trait. In particular, a communication system employed to transmit information about deviations from an isochronous pulse would evolve toward levels of precision comparable between production and perception (Merker, 1999, 2000). This comparable precision is exactly what can be observed in human motoric and perceptual isochrony (Madison and Merker, 2002; Merker et al., 2009), offering some preliminary, indirect support for a possible communicative function of isochrony.

Isochrony does not appear to be used in the overt communication of modern humans, but might have played a role in some form of communication employed by our ancestors. In fact, isochrony is the optimal way to establish synchronized group signaling because it makes the duration of next interval perfectly predictable by another person or conspecific (Merker et al., 2009). This musical perspective on the evolution of isochrony connects to turn-taking, which is a crucial component of human language (Figure 8). Turn-taking allows speakers to effectively interact in conversation: it avoids that speakers’ utterances overlap, while still enabling utterances to occur within a reasonable amount of time from each other. Interestingly, turn-taking in language is both predictive and exogenous, but seems to lack isochrony, except maybe in a few special cases. Still, turn-taking exhibits a particular temporal structure (Stivers et al., 2009; Levinson and Torreira, 2015). This structure appears to arise by a constant 200 ms lag (Figure 9C) between the ends and starts of utterances across cultures (Stivers et al., 2009), rather than a lag between the starts of consecutive utterances. This fixed-interval delay contrasts with the slightly positive or negative lags found in animal synchronization experiments (Figures 9A,B), and the anticipatory reaction in human musical synchronization. So, in modern humans, turn-taking is far from isochrony (except for when it is a by product of utterances having the same duration within and between speakers), but it might promote isochrony (Schultz et al., 2016). This makes turn-taking in modern organisms a potential approach to understand the evolution of isochrony (see Figure 8).

Recognizing speech from analysis of neural activity

Herff C, Heger D, de Pesters A, Telaar D, Brunner P, Schalk G and Schultz T (2015) Brain-to-text: decoding spoken phrases from phone representations in the brain. Front. Neurosci. 9:217. doi: 10.3389/fnins.2015.00217

Abstract

It has long been speculated whether communication between humans and machines based on natural speech related cortical activity is possible. Over the past decade, studies have suggested that it is feasible to recognize isolated aspects of speech from neural signals, such as auditory features, phones or one of a few isolated words. However, until now it remained an unsolved challenge to decode continuously spoken speech from the neural substrate associated with speech and language processing. Here, we show for the first time that continuously spoken speech can be decoded into the expressed words from intracranial electrocorticographic (ECoG) recordings.Specifically, we implemented a system, which we call Brain-To-Text that models single phones, employs techniques from automatic speech recognition (ASR), and thereby transforms brain activity while speaking into the corresponding textual representation. Our results demonstrate that our system can achieve word error rates as low as 25% and phone error rates below 50%. Additionally, our approach contributes to the current understanding of the neural basis of continuous speech production by identifying those cortical regions that hold substantial information about individual phones. In conclusion, the Brain-To- Text system described in this paper represents an important step toward human-machine communication based on imagined speech.

The conclusion

Decoding overt speech production is a necessary first step toward human-computer interaction through imagined speech processes. Our results show that with a limited set of words in the dictionary, Brain-to-Text reconstructs spoken phrases from neural data. The computational phone models in combination with language information make it possible to reconstruct words in unseen spoken utterances solely based on neural signals (see Supplementary Video). Despite the fact that the evaluations in this article have been performed offline, all processing steps of Brain-to-Text and the decoding approach are well suited for eventual real-time online application on desktop computers. The approach introduced here may have important implications for the design of novel brain-computer interfaces, because it may eventually allow people to communicate solely based on brain signals associated with natural language function and with scalable vocabularies.

H/t José Angel García Landa.

How dynamic is word meaning?

3:AM Magazine interviews philosopher Peter Ludlow:

3:AM: In your forthcoming book Living Words: Meaning Underdetermination and the Dynamic Lexicon you’re interested in how elements of meaning that can change radically on a conversation-by-conversation basis. Meanings are dynamic. This is not how language is traditionally thought about is it. To see how different your view is could you say something about the view your arguments challenge?

PL: ... The traditional view is that languages are fairly stable objects that we learn with varying degrees of success and that we then deploy for expressing thoughts and performing certain tasks (by giving orders, instructions, etc). Sometimes the traditional view uses the metaphor of language as a widely shared common currency that agents use to communicate, with individuals words being the common coins of the realm. These common coins are supposed to be more or less fixed; as Locke argued, even Augustus, though he ruled the world, was unable to coin new Latin words.

On my view we are coining new terms all the time – if only for a limited run – and more importantly we are also working together modulating word meanings so as to build microlanguages. These are possibly one-off languages that we construct based on communicative needs. So there is no language that is given to us or that is stable. Languages are things we are constantly building and modifying.

3:AM: I guess the traditional view accommodates some degrees of flex. You’re arguing for a flex that destroys that position though aren’t you?

PL: Yes that’s right. Many people believe that language is context sensitive and somewhat dynamic but I think that even Wittgensteinians failed to appreciate the full nature of the context sensitivity. I argue that word meanings are radically dynamic and that a large part of the conversations we enter into involve the modulations of word meaning. So, to use an example from the linguist Chris Barker, when I say “Smith is tall” I may not be making a claim about the world or even trying to do something in the world, but rather I am trying to modulate the meaning of ‘tall’. If I fix Smith as a safe case of a tall person I’ve possibly broadened the meaning or at least firmed it up for us.

Timing, and speaking

Salon has an interesting article about time and timing (excerpted from, Time Warped: Unlocking the Mysteries of Time Perception, a book by Claudia Hammond). Here's a passage about the time scale involved in speech:

To produce and understand speech, we rely on critical timings of less than a tenth of a second. The difference between the sound of a ‘pa’ and a ‘ba’ is all in the timing of the delay before the subsequent vowel, so if the delay is longer you hear a ‘p’, if it’s short you hear a ‘b.’ If you put your hand on your vocal cords you can even feel that with the ‘ba’ your lips open at the same time as you feel your cords start to vibrate. With the ‘pa’ the vibration starts a moment later. This relies on timing accurate to the millisecond. Even the timing between syllables can be crucial to a phrase’s meaning. With Jimi Hendrix’s lyric, “Excuse me while I kiss the sky,” just a fraction of a second difference in timing is what gives you the famous monde-green, “Excuse me while I kiss this guy.”

There's evidence that this requires that speaker and listener must be muturally synchronized. Their nervous systems are coupled into a single oscillating system.

Thought as Inner Speech

What is thought? It depends, of course, on what you mean by thought. One might use “thought” in the sense of, say, one of three major categories of mental activity: thought, perception, and feeling. Or maybe willing makes it four and perhaps something else makes it five. Whatever. But if you mean thought in that deep and fundamental sense, then it’s a difficult question and I’ll pass on it.

What I have in mind is something less rigorous. I’m interested in the commonsense notion of thought and that, I believe, is more or less inner speech. As such, it’s something that humans do but animals do not. We have language, they do not, hence they cannot have inner speech. On the other hand, animals might well think in the deeper sense I alluded to in the first paragraph.

It’s this common sense notion of thought as inner speech that interests me. I trace my position on this to Lev Semenovich Vygotsky, a Soviet psychologist from the first half of the 20th Century. For a contemporary account in that tradition, see Sydney Lamb’s Pathways of the Brain (1998, pp. 181 ff.).

The rest of this post consists of an account of inner speech that I gave in Beethoven’s Anvil (Basic Books, pp. 151-153). I’m interested in inner speech because it’s a vehicle that allows us to take command of our actions.

* * * * *

What does it mean to say that you cease to think? It means, I believe, that inner speech ceases to play a role in directing your activities. I am thus identifying the commonsense notion of “thinking” with inner speech. Your brain certainly does not shut down when you stop thinking yet remain fully awake, attentive, and performing music. All that ceases is one process.

That process was investigated by Lev Semenovich Vygotsky during the 20s and 30s in the Soviet Union and published in 1934 in his classic Thought and Language. The book was suppressed in 1936 and was not readily available until a decade after WWII. Vygotsky’s general idea is that as others direct the child's actions and perceptions through language, so the child comes to use language in directing her own activities.

Brains Couple When People Talk

Uri Hasson, a Princeton psychologist who's also done some interesting work on brain activity while watching movies, has discovered that, when two people converse, their brains become coupled. Writing in a blog at Scientific America Douglas Fields reports:

There have been many functional brain-imaging studies involving language, but never before have researchers examined both the speaker's and the listener's brains while they communicate to see what is happening inside each brain. The researchers found that when the two people communicate, neural activity over wide regions of their brains becomes almost synchronous, with the listener's brain activity patterns mirroring those sweeping through the speaker's brain, albeit with a short lag of about one second. If the listener, however, fails to comprehend what the speaker is trying to communicate, their brain patterns decouple.

. . . .

In order to find out what happens in the brain when the speaker and listener communicate or fail to connect, Hasson, an assistant professor in Princeton's Department of Psychology, and his team had to first overcome both technical problems using new analytical methods as well as special nonmagnetic noise-canceling microphones. He asked his student to tell an unrehearsed simple story while imaging her brain. Then they played back that story to several listeners and found that the listener's brain patterns closely matched what was happening inside the speaker's head as she told the story.

These results are exciting but not surprising. Back in the late 60s and early 1970s William Condon did high-speed video taping of people interacting with one another. He found, for example, that the listener's head and body movements tracked the intonation patters of the speaker's language. Interestingly enough, this was true even for neonates, their body motions tracked speech patterns of nearby speakers.

I made such interactional synchrony the conceptual centerpiece of my 2001 book on music, Beethoven's Anvil. I also reprise and extend some of those ideas in my essay-review of Steven Mithen's The Singing Neanderthals. See also my post, The Sound of Many Hands Clapping: Group Intentionality.