Synthesizing Proteins on the Graphics Card. Protein Folding and the Limits of Critical AI Studies

TL;DR

This paper critically examines the use of transformer architectures in protein folding, highlighting their non-linguistic processing and the implications for understanding AI's role in scientific knowledge creation.

Contribution

It challenges the language paradigm in computational biology and reveals the non-linguistic, epistemological aspects of transformer-based protein folding models.

Findings

Transformers utilize non-linguistic, high-dimensional vector representations.

The analogy between language and proteins is historically and conceptually limited.

Transformer architecture creates a new epistemological space in scientific knowledge-making.

Abstract

This paper investigates the application of the transformer architecture in protein folding, as exemplified by DeepMind's AlphaFold project, and its implications for the understanding of so-called large language models. The prevailing discourse often assumes a ready-made analogy between proteins, encoded as sequences of amino acids, and natural language, which we term the language paradigm of computational (structural) biology. Instead of assuming this analogy as given, we critically evaluate it to assess the kind of knowledge-making afforded by the transformer architecture. We first trace the analogy's emergence and historical development, carving out the influence of structural linguistics on structural biology beginning in the mid-20th century. We then examine three often overlooked preprocessing steps essential to the transformer architecture, including subword tokenization, word embedding, and positional encoding, to demonstrate its regime of representation based on continuous, high-dimensional vector spaces, which departs from the discrete nature of language. The successful deployment of transformers in protein folding, we argue, discloses what we consider a non-linguistic approach to token processing intrinsic to the architecture. We contend that through this non-linguistic processing, the transformer architecture carves out unique epistemological territory and produces a new class of knowledge, distinct from established domains. We contend that our search for intelligent machines has to begin with the shape, rather than the place, of intelligence. Consequently, the emerging field of critical AI studies should take methodological inspiration from the history of science in its quest to conceptualize the contributions of artificial intelligence to knowledge-making, within and beyond the domain-specific sciences.