Generalization or Hallucination? Understanding Out-of-Context Reasoning in Transformers

TL;DR

This paper investigates out-of-context reasoning in large language models, revealing it as a key factor behind both their ability to generalize from new facts and their tendency to hallucinate, supported by theoretical and empirical analysis.

Contribution

It introduces a formal framework for out-of-context reasoning, demonstrating how model architecture and implicit bias influence this phenomenon.

Findings

OCR drives both generalization and hallucination depending on causal relations

A simple attention-only transformer can learn OCR in a synthetic task

Gradient descent bias explains the model's ability to associate facts efficiently

Abstract

Large language models (LLMs) can acquire new knowledge through fine-tuning, but this process exhibits a puzzling duality: models can generalize remarkably from new facts, yet are also prone to hallucinating incorrect information. However, the reasons for this phenomenon remain poorly understood. In this work, we argue that both behaviors stem from a single mechanism known as out-of-context reasoning (OCR): the ability to deduce implications by associating concepts, even those without a causal link. Our experiments across five prominent LLMs confirm that OCR indeed drives both generalization and hallucination, depending on whether the associated concepts are causally related. To build a rigorous theoretical understanding of this phenomenon, we then formalize OCR as a synthetic factual recall task. We empirically show that a one-layer single-head attention-only transformer with factorized output and value matrices can learn to solve this task, while a model with combined weights cannot, highlighting the crucial role of matrix factorization. Our theoretical analysis shows that the OCR capability can be attributed to the implicit bias of gradient descent, which favors solutions that minimize the nuclear norm of the combined output-value matrix. This mathematical structure explains why the model learns to associate facts and implications with high sample efficiency, regardless of whether the correlation is causal or merely spurious. Ultimately, our work provides a theoretical foundation for understanding the OCR phenomenon, offering a new lens for analyzing and mitigating undesirable behaviors from knowledge injection.