In-context learning emerges in chemical reaction networks without attention

TL;DR

This paper demonstrates that chemical reaction networks can perform in-context learning through a mechanism called subspace projection, expanding the understanding of computation in chemical and biological systems.

Contribution

It introduces a novel mechanism for in-context learning in chemical systems, bypassing the need for attention mechanisms used in neural networks.

Findings

Chemical processes can achieve in-context learning via subspace projection.

Performance is robust to input encoding and dynamical choices.

The number of tunable degrees of freedom limits performance.

Abstract

We investigate whether chemical processes can perform in-context learning (ICL), a mode of computation typically associated with transformer architectures. ICL allows a system to infer task-specific rules from a sequence of examples without relying solely on fixed parameters. Traditional ICL relies on a pairwise attention mechanism which is not obviously implementable in chemical systems. However, we show theoretically and numerically that chemical processes can achieve ICL through a mechanism we call subspace projection, in which the entire input vector is mapped onto comparison subspaces, with the dominant projection determining the computational output. We illustrate this mechanism analytically in small chemical systems and show numerically that performance is robust to input encoding and dynamical choices, with the number of tunable degrees of freedom in the input encoding as a key limitation. Our results provide a blueprint for realizing ICL in chemical or other physical media and suggest new directions for designing adaptive synthetic chemical systems and understanding possible biological computation in cells.