StreetMath: Study of LLMs' Approximation Behaviors

TL;DR

This paper introduces StreetMath, a benchmark for evaluating large language models' ability to perform approximate reasoning in informal mathematical tasks, revealing their tendencies to seek exact solutions and use separate components for different operations.

Contribution

The work presents a new benchmark for approximation in LLMs, extensive evaluations across multiple models, and mechanistic insights into their internal computation behaviors.

Findings

LLMs often attempt exact calculations or external tool use in approximation tasks.

Models sometimes produce correct answers early in processing but consume more tokens.

Exact and approximate arithmetic rely on largely separate neural components.

Abstract

There is a substantial body of literature examining the mathematical reasoning capabilities of large language models (LLMs), particularly their performance on precise arithmetic operations in autoregressive architectures. However, their ability to perform approximate reasoning in informal, fast-paced mathematical operations has received far less attention, especially among non-autoregressive decoder models. Our work addresses this gap by introducing StreetMath, a benchmark designed to evaluate models' approximation abilities under real-world approximation scenarios. We conduct extensive evaluations across different LLM architectures: Qwen3-4B-Instruct-2507, Qwen3-4B-Thinking-2507, Dream-v0-Instruct-7B, Falcon-Mamba-7B-Instruct, and Mamba-GPT-3B. Furthermore, we apply mechanistic interpretability techniques to probe their internal computational states. Our analysis reveals that LLMs generally attempt to compute exact values or invoke external tools even in tasks that call for approximation. Moreover, while models sometimes reach the correct answer in early layers or steps, they still consume more tokens when solving approximation tasks. Additional experiments indicate that exact and approximate arithmetic operations rely on largely separate neural components. Drawing upon research on cognitive psychology, we argue that LLMs do not exhibit cognitive miserliness in the same way humans do in street math settings. We open source our work https://github.com/ctseng777/StreetMath