Hallucination, Monofacts, and Miscalibration: An Empirical Investigation

TL;DR

This paper empirically investigates the relationship between hallucinations, monofacts, and miscalibration in language models, introducing techniques to control and reduce hallucinations while analyzing their trade-offs.

Contribution

It provides the first empirical analysis of the three-way relationship in classical and transformer models, and introduces selective upweighting to reduce hallucinations.

Findings

Monofact rate positively correlates with hallucination.

Selective upweighting reduces hallucination by up to 40%.

Trade-off observed between accuracy and hallucination reduction.

Abstract

Hallucinated facts in large language models (LLMs) have recently been shown to obey a statistical lower bound determined by the monofact rate (related to the classical Good-Turing missing mass estimator) minus model miscalibration (Kalai & Vempala, 2024). We present the first empirical investigation of this three-way relationship in classical n-gram models and fine-tuned encoder-decoder Transformers. By generating training data from Pareto distributions with varying shape parameters, we systematically control the monofact rates and establish its positive relationship with hallucination. To bridge theory and practice, we derive an empirical analog of the hallucination bound by replacing the population miscalibration term (Section 2.1) with an empirical bin-wise KL divergence and confirm its practical viability. We then introduce selective upweighting -- a simple yet effective technique that strategically repeats as little as 5% of training examples -- to deliberately inject miscalibration into the model. This intervention reduces hallucination by up to 40%, challenging universal deduplication policies. Our experiments reveal a critical trade-off: selective upweighting maintains pre-injection levels of accuracy while substantially reducing hallucination, whereas standard training gradually improves accuracy but fails to address persistently high hallucination, indicating an inherent tension in optimization objectives.