LLMs are not (consistently) Bayesian: Quantifying internal (in)consistencies of LLMs' probabilistic beliefs

TL;DR

This paper investigates how large language models update their probabilistic beliefs, revealing that many do not follow Bayesian principles and often perform better with heuristic updates, highlighting issues in their belief modeling.

Contribution

It introduces a novel method to quantify internal inconsistencies in LLMs' belief updates and compares Bayesian versus heuristic approaches across various experiments.

Findings

Non-Bayesian heuristics often outperform Bayesian updates in downstream tasks.

LLMs' belief updates are frequently inconsistent with Bayesian principles.

The proposed measure can diagnose issues in LLM-based inference systems.

Abstract

Modern AI systems are being deployed in complex domains such as medicine, science, and law, where it is important that they not only produce correct answers, but also represent and update uncertain beliefs about the world as new evidence arrives. We introduce the novel technique of studying LLMs as information processing rules and utilize the information processing gap to study the internal (in)consistencies of how LLMs update their probabilistic beliefs from evidence. Our extensive experiments evaluate multiple approaches in which LLMs can incorporate evidence into their beliefs. Some of these approaches produce (nearly) Bayesian updates; others seem to use a learned heuristic. Surprisingly, the non-Bayesian heuristic updates often outperform exact Bayesian computation in terms of downstream task performance -- indicating the LLMs' probabilistic models of the world are misspecified. Lastly, we show how our measure can provide diagnostics to identify issues with LLM-powered inferential systems.