Non-literal Understanding of Number Words by Language Models

TL;DR

This paper examines whether large language models interpret number words non-literally like humans, identifying key differences in pragmatic reasoning and proposing chain-of-thought prompting to improve human-like understanding.

Contribution

It introduces a framework for analyzing LLMs' pragmatic reasoning about numbers and demonstrates how chain-of-thought prompting can align their interpretations more closely with human cognition.

Findings

LLMs diverge from human pragmatic interpretation of numbers

Decomposition of reasoning reveals where LLMs differ from humans

Chain-of-thought prompting improves LLMs' human-like understanding

Abstract

Humans naturally interpret numbers non-literally, effortlessly combining context, world knowledge, and speaker intent. We investigate whether large language models (LLMs) interpret numbers similarly, focusing on hyperbole and pragmatic halo effects. Through systematic comparison with human data and computational models of pragmatic reasoning, we find that LLMs diverge from human interpretation in striking ways. By decomposing pragmatic reasoning into testable components, grounded in the Rational Speech Act framework, we pinpoint where LLM processing diverges from human cognition -- not in prior knowledge, but in reasoning with it. This insight leads us to develop a targeted solution -- chain-of-thought prompting inspired by an RSA model makes LLMs' interpretations more human-like. Our work demonstrates how computational cognitive models can both diagnose AI-human differences and guide development of more human-like language understanding capabilities.