Hallucination Basins: A Dynamic Framework for Understanding and Controlling LLM Hallucinations

TL;DR

This paper introduces a geometric dynamical systems framework to understand LLM hallucinations, revealing task-dependent basin structures in latent space and proposing geometry-aware steering to reduce hallucinations.

Contribution

It formalizes the behavior of hallucinations with task-complexity and multi-basin theorems, and demonstrates geometry-aware steering as a method to mitigate hallucinations without retraining.

Findings

Factoid tasks show clearer basin separation.

Summarization tasks have overlapping basin structures.

Geometry-aware steering reduces hallucination probability.

Abstract

Large language models (LLMs) hallucinate: they produce fluent outputs that are factually incorrect. We present a geometric dynamical systems framework in which hallucinations arise from task-dependent basin structure in latent space. Using autoregressive hidden-state trajectories across multiple open-source models and benchmarks, we find that separability is strongly task-dependent rather than universal: factoid settings can show clearer basin separation, whereas summarization and misconception-heavy settings are typically less stable and often overlap. We formalize this behavior with task-complexity and multi-basin theorems, characterize basin emergence in L-layer transformers, and show that geometry-aware steering can reduce hallucination probability without retraining.