The geometry of intelligence: From protein folding to language and the epistemic affordances of the transformer architecture

Fabian Offert, Paul Kim, Qiaoyu Cai, Synthesizing Proteins on the Graphics Card. Protein Folding and the Limits of Critical AI Studies, arXiv:2405.09788v1

Abstract: This paper investigates the application of the transformer architecture in protein folding, as exemplified by DeepMind’s AlphaFold project, and its implications for the understanding of large language models as models of language. The prevailing discourse often assumes a ready-made analogy between proteins – encoded as sequences of amino acids – and natural language – encoded as sequences of discrete symbols. Instead of assuming as given the linguistic structure of proteins, we critically evaluate this analogy to assess the kind of knowledge-making afforded by the transformer architecture. We first trace the analogy’s emergence and historical development, carving out the influence of structural linguistics on structural biology beginning in the mid-20th century. We then examine three often overlooked pre-processing steps essential to the transformer architecture, including subword tokenization, word embedding, and positional encoding, to demonstrate its regime of representation based on continuous, high-dimensional vector spaces, which departs from the discrete, semantically demarcated symbols of language. The successful deployment of transformers in protein folding, we argue, discloses what we consider a non-linguistic approach to token processing intrinsic to the architecture. We contend that through this non-linguistic processing, the transformer architecture carves out unique epistemological territory and produces a new class of knowledge, distinct from established domains. We contend that our search for intelligent machines has to begin with the shape, rather than the place, of intelligence. Consequently, the emerging field of critical AI studies should take methodological inspiration from the history of science in its quest to conceptualize the contributions of artificial intelligence to knowledge-making, within and beyond the domain-specific sciences.

Think of this in relation to the Structured Physical System Hypothesis (SPSH) of Saty Chary:

‘A structured physical system has the necessary and sufficient means for specific intelligent response’. By structured physical system, I mean, an analog design, e.g. a Rube Goldberg apparatus, or a Braitenberg (!) vehicle, etc. This is in contrast to this: PSSH - Physical Symbol System Hypothesis - 'A physical symbol system has the necessary and sufficient means for general intelligent action'.