Language Models Do Hard Arithmetic Tasks Easily and Hardly Do Easy Arithmetic Tasks

TL;DR

Large language models can predict the first digit of complex multiplication tasks accurately without reasoning, but struggle with the last digit unless conditioned on higher digits, revealing strengths and limitations in their arithmetic capabilities.

Contribution

This paper uncovers the contrasting abilities of LLMs in predicting different digits of multiplication tasks and demonstrates improved last-digit prediction when conditioned on higher digits.

Findings

LLMs predict first digits of complex multiplications accurately without chain of thought.

LLMs often fail to predict last digits correctly in simple multiplication tasks.

Conditioning on higher digits significantly improves last-digit prediction confidence.

Abstract

The ability (and inability) of large language models (LLMs) to perform arithmetic tasks has been the subject of much theoretical and practical debate. We show that LLMs are frequently able to correctly and confidently predict the first digit of n-digit by m-digit multiplication tasks without using chain of thought reasoning, despite these tasks require compounding operations to solve. Simultaneously, LLMs in practice often fail to correctly or confidently predict the last digit of an n-digit by m-digit multiplication, a task equivalent to 1-digit by 1-digit multiplication which can be easily learned or memorized. We show that the latter task can be solved more robustly when the LLM is conditioned on all of the correct higher-order digits, which on average increases the confidence of the correct last digit on 5-digit by 5-digit multiplication tasks using Llama 2-13B by over 230% (0.13 to 0.43) and Mistral-7B by 150% (0.22 to 0.55).