Fast weight programming and linear transformers: from machine learning to neurobiology

TL;DR

This paper reviews fast weight programmers (FWPs), a class of neural networks with dynamic 2D matrix hidden states, exploring their computational properties, connections to transformers, and relevance to biological synaptic plasticity.

Contribution

It provides a comprehensive overview of FWPs, their theoretical foundations, and their links to both machine learning models and neurobiological processes.

Findings

FWPs use dynamic 2D matrix hidden states for short-term memory.

Connections between FWPs, transformers, and state space models are discussed.

Potential biological relevance of FWPs to synaptic plasticity is explored.

Abstract

Recent advances in artificial neural networks for machine learning, and language modeling in particular, have established a family of recurrent neural network (RNN) architectures that, unlike conventional RNNs with vector-form hidden states, use two-dimensional (2D) matrix-form hidden states. Such 2D-state RNNs, known as Fast Weight Programmers (FWPs), can be interpreted as a neural network whose synaptic weights (called fast weights) dynamically change over time as a function of input observations, and serve as short-term memory storage; corresponding synaptic weight modifications are controlled or programmed by another network (the programmer) whose parameters are trained (e.g., by gradient descent). In this Primer, we review the technical foundations of FWPs, their computational characteristics, and their connections to transformers and state space models. We also discuss connections between FWPs and models of synaptic plasticity in the brain, suggesting a convergence of natural and artificial intelligence.