Investigating the interaction of linguistic and mathematical reasoning in language models using multilingual number puzzles

TL;DR

This paper explores why large language models struggle with multilingual number puzzles, revealing that explicit mathematical symbols are crucial and that models lack the implicit numeral structure understanding humans use.

Contribution

The study uncovers the importance of explicit symbols for mathematical reasoning in LLMs and highlights their inability to infer implicit numeral structures from language data.

Findings

Models require explicit mathematical symbols to solve number puzzles.

LLMs lack the ability to infer implicit numeral structures.

Humans use linguistic cues to understand number composition, unlike current models.

Abstract

Across languages, numeral systems vary widely in how they construct and combine numbers. While humans consistently learn to navigate this diversity, large language models (LLMs) struggle with linguistic-mathematical puzzles involving cross-linguistic numeral systems, which humans can learn to solve successfully. We investigate why this task is difficult for LLMs through a series of experiments that untangle the linguistic and mathematical aspects of numbers in language. Our experiments establish that models cannot consistently solve such problems unless the mathematical operations in the problems are explicitly marked using known symbols ($+$, $\times$, etc., as in "twenty + three"). In further ablation studies, we probe how individual parameters of numeral construction and combination affect performance. While humans use their linguistic understanding of numbers to make inferences about the implicit compositional structure of numerals, LLMs seem to lack this notion of implicit numeral structure. We conclude that the ability to flexibly infer compositional rules from implicit patterns in human-scale data remains an open challenge for current reasoning models.