Self-Consistency of Large Language Models under Ambiguity

TL;DR

This paper evaluates the self-consistency of large language models in ambiguous tasks, revealing that models tend to be internally multi-possibility aware and that self-consistency improves with capability, despite calibration issues.

Contribution

The authors introduce a benchmark for assessing self-consistency in LLMs under ambiguity and analyze models' behavior, robustness, and internal probability distributions.

Findings

Models achieve 67-82% consistency, higher than random chance.

Self-consistency increases with model capability.

Models often assign significant probability to alternative answers.

Abstract

Large language models (LLMs) that do not give consistent answers across contexts are problematic when used for tasks with expectations of consistency, e.g., question-answering, explanations, etc. Our work presents an evaluation benchmark for self-consistency in cases of under-specification where two or more answers can be correct. We conduct a series of behavioral experiments on the OpenAI model suite using an ambiguous integer sequence completion task. We find that average consistency ranges from 67\% to 82\%, far higher than would be predicted if a model's consistency was random, and increases as model capability improves. Furthermore, we show that models tend to maintain self-consistency across a series of robustness checks, including prompting speaker changes and sequence length changes. These results suggest that self-consistency arises as an emergent capability without specifically training for it. Despite this, we find that models are uncalibrated when judging their own consistency, with models displaying both over- and under-confidence. We also propose a nonparametric test for determining from token output distribution whether a model assigns non-trivial probability to alternative answers. Using this test, we find that despite increases in self-consistency, models usually place significant weight on alternative, inconsistent answers. This distribution of probability mass provides evidence that even highly self-consistent models internally compute multiple possible responses.