Language Models are Symbolic Learners in Arithmetic

TL;DR

This paper investigates whether language models learn true arithmetic algorithms or rely on superficial pattern shortcuts, revealing they primarily learn hierarchical symbolic shortcuts rather than genuine computation.

Contribution

The paper introduces subgroup induction to analyze LM arithmetic learning, demonstrating models rely on simple shortcuts and hierarchical symbol mappings instead of algorithms.

Findings

LMs exhibit a U-shaped accuracy pattern in multi-digit multiplication

Accuracy correlates with the quality of simple, low-token subgroups

Models learn a hierarchy of symbolic shortcuts rather than true algorithms

Abstract

The prevailing question in LM performing arithmetic is whether these models learn to truly compute or if they simply master superficial pattern matching. In this paper, we argues for the latter, presenting evidence that LMs act as greedy symbolic learners, prioritizing the simplest possible shortcuts to fit the stats of dataset to solve arithmetic tasks. To investigate this, we introduce subgroup induction, a practical framework adapted from Solomonoff Induction (SI), one of the most powerful universal predictors. Our framework analyzes arithmetic problems by breaking them down into subgroups-minimal mappings between a few input digits and a single output digit. Our primary metric, subgroup quality, measures the viability of these shortcuts. Experiments reveal a distinct U-shaped accuracy pattern in multi-digit multiplication: LMs quickly master the first and last output digits while struggling with those in the middle. We demonstrate this U-shape is not coincidental; it perfectly mirrors the quality of the simplest possible subgroups, those requiring the fewest input tokens. This alignment suggests a core learning mechanism: LMs first learn easy, low-token shortcuts and only incorporate more complex, multi-token patterns as training progresses. They do not learn the algorithm of multiplication but rather a hierarchy of increasingly complex symbol-to-symbol mappings. Ultimately, our findings suggest that the path to arithmetic mastery for LMs is not paved with algorithms, but with a cascade of simple, hierarchically-learned symbolic shortcuts.