Uncovering a Universal Abstract Algorithm for Modular Addition in Neural Networks

TL;DR

This paper proposes a universal abstract algorithm for modular addition in neural networks, demonstrating that diverse solutions are unified under the approximate Chinese Remainder Theorem across different architectures.

Contribution

It introduces a testable universality hypothesis and provides the first theory-backed interpretation of multilayer networks solving modular addition.

Findings

Neural networks implement the approximate Chinese Remainder Theorem for modular addition.

Neurons activate exclusively on approximate cosets.

Universal solutions in deep networks require only O(log n) features.

Abstract

We propose a testable universality hypothesis, asserting that seemingly disparate neural network solutions observed in the simple task of modular addition are unified under a common abstract algorithm. While prior work interpreted variations in neuron-level representations as evidence for distinct algorithms, we demonstrate - through multi-level analyses spanning neurons, neuron clusters, and entire networks - that multilayer perceptrons and transformers universally implement the abstract algorithm we call the approximate Chinese Remainder Theorem. Crucially, we introduce approximate cosets and show that neurons activate exclusively on them. Furthermore, our theory works for deep neural networks (DNNs). It predicts that universally learned solutions in DNNs with trainable embeddings or more than one hidden layer require only O(log n) features, a result we empirically confirm. This work thus provides the first theory-backed interpretation of multilayer networks solving modular addition. It advances generalizable interpretability and opens a testable universality hypothesis for group multiplication beyond modular addition.