Math Takes Two: A test for emergent mathematical reasoning in communication

TL;DR

Math Takes Two introduces a benchmark to evaluate whether language models can develop emergent mathematical reasoning through communication without prior knowledge, focusing on discovering shared symbolic protocols from scratch.

Contribution

The paper presents a novel benchmark that assesses emergent mathematical reasoning by enabling agents to develop shared symbolic protocols without predefined mathematical language.

Findings

Agents can develop shared symbolic protocols for mathematical tasks.

The benchmark reveals the models' ability to discover latent structures from scratch.

Math Takes Two offers a new way to evaluate emergent numerical reasoning.

Abstract

Although language models demonstrate remarkable proficiency on mathematical benchmarks, it remains unclear whether this reflects true mathematical reasoning or statistical pattern matching over learning formal syntax. Most existing evaluations rely on symbolic problems grounded in established mathematical conventions, limiting insight into the models' ability to construct abstract concepts from first principles. In this work, we propose Math Takes Two, a new benchmark designed to assess the emergence of mathematical reasoning through communication. Motivated by the hypothesis that mathematical cognition in humans co-evolved with the need for precise communication, our benchmark tests whether two agents, without prior mathematical knowledge, can develop a shared symbolic protocol to solve a visually grounded task where the use of a numerical system facilitates extrapolation. Unlike many current datasets, our benchmark eschews predefined mathematical language, instead requiring agents to discover latent structure and representations from scratch. Math Takes Two thus provides a novel lens through which to develop and evaluate models with emergent numerical reasoning capabilities.