Arithmetic in the Wild: Llama uses Base-10 Addition to Reason About Cyclic Concepts

TL;DR

This paper investigates how Llama-3.1-8B performs cyclic reasoning by reusing a base-10 addition mechanism and Fourier features, revealing task-agnostic computation and neuron specialization.

Contribution

It uncovers that Llama-3.1-8B employs a universal addition process with Fourier features and identifies specific neurons dedicated to different cyclic sums.

Findings

Llama-3.1-8B uses base-10 addition to reason over cyclic concepts.

Fourier features with periods like 2, 5, and 10 are used for sum computation.

A sparse set of 28 neurons encodes different cyclic sum computations.

Abstract

Does structure in representations imply structure in computation? We study how Llama-3.1-8B reasons over cyclic concepts (e.g., "what month is six months after August?"). Even though Llama-3.1-8B's representations for these concepts are circularly structured, we find that instead of directly computing modular addition in the period of the cyclic concept (e.g., 12 for months), the model re-uses a generic addition mechanism across tasks that operates independently of concept-specific geometry. First, it computes the sum of its two inputs using base-10 addition (six + August=14). Then, it maps this sum back to cyclic concept space (14->February). We show that Llama-3.1-8B uses task-agnostic Fourier features to compute these sums--in fact, these features have periods that respect standard base-10 addition, e.g., 2, 5, and 10, rather than the cyclic concept period (e.g., 12 for months). Furthermore, we identify a sparse set of 28 MLP neurons re-used across all tasks (approximately 0.2% of the MLP at layer 18) that can be partitioned into disjoint clusters, each computing the sum for a Fourier feature with a different period. Our work highlights how an interplay between causal abstraction and feature geometry can deepen our mechanistic understanding of LMs.