The Effect of Scripts and Formats on LLM Numeracy

TL;DR

This paper investigates how large language models handle various numeral scripts and formats, revealing accuracy drops with underrepresented styles and proposing prompting strategies to improve numerical reasoning across diverse representations.

Contribution

It identifies the impact of numeral script and format variations on LLM numeracy and proposes prompting techniques to mitigate these challenges.

Findings

Accuracy drops with underrepresented numeral scripts and formats

Few-shot prompting improves numerical reasoning performance

Explicit numeral mapping narrows the performance gap

Abstract

Large language models (LLMs) have achieved impressive proficiency in basic arithmetic, rivaling human-level performance on standard numerical tasks. However, little attention has been given to how these models perform when numerical expressions deviate from the prevailing conventions present in their training corpora. In this work, we investigate numerical reasoning across a wide range of numeral scripts and formats. We show that LLM accuracy drops substantially when numerical inputs are rendered in underrepresented scripts or formats, despite the underlying mathematical reasoning being identical. We further demonstrate that targeted prompting strategies, such as few-shot prompting and explicit numeral mapping, can greatly narrow this gap. Our findings highlight an overlooked challenge in multilingual numerical reasoning and provide actionable insights for working with LLMs to reliably interpret, manipulate, and generate numbers across diverse numeral scripts and formatting styles.