On the Limits of Language Generation: Trade-Offs Between Hallucination and Mode Collapse

TL;DR

This paper investigates the fundamental trade-offs in language generation models between avoiding hallucinations and mode collapse, showing that achieving both simultaneously is generally impossible without negative feedback, but can be improved with it.

Contribution

It provides theoretical bounds and impossibility results for language models to generate all unseen strings, highlighting the importance of negative examples to mitigate hallucination and mode collapse.

Findings

Generation with breadth is impossible for most models without negative examples.

Consistent generation without breadth is possible for countable language collections.

Negative examples can enable models to generate all unseen strings, reducing hallucinations.

Abstract

Specifying all desirable properties of a language model is challenging, but certain requirements seem essential. Given samples from an unknown language, the trained model should produce valid strings not seen in training and be expressive enough to capture the language's full richness. Otherwise, outputting invalid strings constitutes "hallucination," and failing to capture the full range leads to "mode collapse." We ask if a language model can meet both requirements. We investigate this within a statistical language generation setting building on Gold and Angluin. Here, the model receives random samples from a distribution over an unknown language K, which belongs to a possibly infinite collection of languages. The goal is to generate unseen strings from K. We say the model generates from K with consistency and breadth if, as training size increases, its output converges to all unseen strings in K. Kleinberg and Mullainathan [KM24] asked if consistency and breadth in language generation are possible. We answer this negatively: for a large class of language models, including next-token prediction models, this is impossible for most collections of candidate languages. This contrasts with [KM24]'s result, showing consistent generation without breadth is possible for any countable collection of languages. Our finding highlights that generation with breadth fundamentally differs from generation without breadth. As a byproduct, we establish near-tight bounds on the number of samples needed for generation with or without breadth. Finally, our results offer hope: consistent generation with breadth is achievable for any countable collection of languages when negative examples (strings outside K) are available alongside positive ones. This suggests that post-training feedback, which encodes negative examples, can be crucial in reducing hallucinations while limiting mode collapse.