Language Model Probabilities are Not Calibrated in Numeric Contexts

TL;DR

This paper investigates whether language models accurately reflect the probabilities of numeric and categorical options in context, finding they are poorly calibrated and exhibit systematic biases influenced by artifacts like word order and frequency.

Contribution

It provides the first systematic analysis of language model calibration in numeric contexts, revealing significant biases and calibration issues even in simple settings.

Findings

Language models are poorly calibrated in numeric contexts.

Systematic biases are influenced by word order, identity, and frequency.

Models do not proportionally assign probabilities to options as expected.

Abstract

Some statements have one well-defined continuation (e.g., "the Eiffel Tower is in [Paris]"), whereas others have a natural distribution over multiple options (e.g., "the weighted coin flip was [Heads/Tails].") We argue that language model (LM) outputs should capture these natural distributions. Our work specifically tests whether LM output probabilities are calibrated to numeric information within their textual contexts. For example, if the context (the prompt) concerns two equally likely options (e.g., heads or tails for a fair coin), the LM output probabilities should also be equal. Likewise, in a context with nonuniformly likely events (e.g., rolling a pair with two dice) an LM should output proportionate probabilities. However, we find that even in simple settings, the best LMs (1) are poorly calibrated and (2) have systematic biases: artifacts like word identity, word order, and word frequency all impact calibration. For example, gpt-4o-mini often picks the first of two options presented in the prompt regardless of the options' implied likelihoods, whereas Llama-3.1-8B picks the second. Models do not allocate probability mass among valid options in a calibrated manner.