Can LLMs subtract numbers?

TL;DR

This paper systematically evaluates large language models' ability to perform subtraction, revealing their struggles with negative results and showing that instruction tuning significantly improves their accuracy in generating correct signs.

Contribution

It provides the first comprehensive analysis of subtraction in LLMs, highlighting their limitations and demonstrating the effectiveness of instruction tuning for improving subtraction accuracy.

Findings

LLMs perform worse on subtraction than addition.

Errors mainly occur when the result should be negative.

Instruction tuning significantly improves negative sign generation.

Abstract

We present a systematic study of subtraction in large language models (LLMs). While prior benchmarks emphasize addition and multiplication, subtraction has received comparatively little attention despite being structurally distinct as a non-commutative operation. We evaluate eight pretrained LLMs spanning four families on addition and subtraction problems. Our experiments reveal that subtraction accuracy lags behind addition by a wide margin. We find that the errors for ($a-b$) are concentrated in cases where ($a<b$). In such cases, LLMs frequently produce the correct magnitude but omit the negative sign. Probing analyses show that LLMs internally encode whether results should be negative, yet this information is often not reflected in generated outputs. We further test well-known techniques such as few-shot learning and instruction-tuning to see if they can improve the LLMs' performance. Our results suggest that while few-shot prompting yields modest gains, the instruction-tuned models achieve near-perfect accuracies in generating the negative sign. Together, these findings provide a clearer characterization of the limitations and recoverability of LLMs' arithmetic capabilities in subtraction.