Exposing Numeracy Gaps: A Benchmark to Evaluate Fundamental Numerical Abilities in Large Language Models

TL;DR

This paper introduces NumericBench, a new benchmark designed to evaluate and expose the weaknesses of large language models in fundamental numerical reasoning tasks, highlighting the need for models to better understand and manipulate numbers.

Contribution

The paper presents NumericBench, a comprehensive benchmark for six core numerical abilities, addressing gaps in existing evaluation methods for LLMs' numerical reasoning skills.

Findings

State-of-the-art LLMs show significant weaknesses in numerical reasoning.

NumericBench reveals persistent numerical understanding gaps in models like GPT-4.

Benchmark datasets include synthetic and real-world data, challenging models with complex tasks.

Abstract

Large Language Models (LLMs) have demonstrated impressive capabilities in natural language processing tasks, such as text generation and semantic understanding. However, their performance on numerical reasoning tasks, such as basic arithmetic, numerical retrieval, and magnitude comparison, remains surprisingly poor. This gap arises from their reliance on surface-level statistical patterns rather than understanding numbers as continuous magnitudes. Existing benchmarks primarily focus on either linguistic competence or structured mathematical problem-solving, neglecting fundamental numerical reasoning required in real-world scenarios. To bridge this gap, we propose NumericBench, a comprehensive benchmark to evaluate six fundamental numerical capabilities: number recognition, arithmetic operations, contextual retrieval, comparison, summary, and logical reasoning. NumericBench includes datasets ranging from synthetic number lists to the crawled real-world data, addressing challenges like long contexts, noise, and multi-step reasoning. Extensive experiments on state-of-the-art LLMs, including GPT-4 and DeepSeek, reveal persistent weaknesses in numerical reasoning, highlighting the urgent need to improve numerically-aware language modeling. The benchmark is released in: https://github.com/TreeAI-Lab/NumericBench.