Semantic reconstruction of continuous language from non-invasive brain recordings

it is generally thought that fMRI responses are too slow for language decoding. to overcome this low temporal resolution, we developed a bayesian decoder that combines state of the art language models and encoding models to generate rapidly changing word sequences (2/7)

— Jerry Tang (@jerryptang) September 30, 2022

the same decoder also worked on brain responses while subjects imagined telling stories, even though the decoder was only trained on perceived speech data. we expect that training the decoder on some imagined speech data will further improve performance (4/7) pic.twitter.com/z63D7Xe3Sa

— Jerry Tang (@jerryptang) September 30, 2022

Abstract of linked article:

A brain-computer interface that decodes continuous language from non-invasive recordings would have many scientific and practical applications. Currently, however, decoders that reconstruct continuous language use invasive recordings from surgically implanted electrodes, while decoders that use non-invasive recordings can only identify stimuli from among a small set of letters, words, or phrases. Here we introduce a non-invasive decoder that reconstructs continuous natural language from cortical representations of semantic meaning recorded using functional magnetic resonance imaging (fMRI). Given novel brain recordings, this decoder generates intelligible word sequences that recover the meaning of perceived speech, imagined speech, and even silent videos, demonstrating that a single language decoder can be applied to a range of semantic tasks. To study how language is represented across the brain, we tested the decoder on different cortical networks, and found that natural language can be separately decoded from multiple cortical networks in each hemisphere. As brain-computer interfaces should respect mental privacy, we tested whether successful decoding requires subject cooperation, and found that subject cooperation is required both to train and to apply the decoder. Our study demonstrates that continuous language can be decoded from non-invasive brain recordings, enabling future multipurpose brain-computer interfaces.