Escaping Mode Collapse in LLM Generation via Geometric Regulation

TL;DR

This paper introduces a geometric regulation method called RMR to mitigate mode collapse in large language models, significantly improving diversity and stability during text generation.

Contribution

The paper presents RMR, a novel online intervention that reduces mode collapse by regulating the model's internal state-space, a perspective not addressed by previous methods.

Findings

RMR substantially reduces mode collapse across multiple large language models.

RMR enables stable, high-quality generation at very low entropy rates.

Standard decoding collapses near 2.0 nats/step, while RMR maintains diversity at 0.8 nats/step.

Abstract

Mode collapse is a persistent challenge in generative modeling and appears in autoregressive text generation as behaviors ranging from explicit looping to gradual loss of diversity and premature trajectory convergence. We take a dynamical-systems view and reinterpret mode collapse as reduced state-space accessibility caused by *geometric collapse*: during generation, the model's internal trajectory becomes confined to a low-dimensional region of its representation space. This implies mode collapse is not purely a token-level phenomenon and cannot be reliably solved by symbolic constraints or probability-only decoding heuristics. Guided by this perspective, we propose *Reinforced Mode Regulation* (RMR), a lightweight, online state-space intervention that regulates dominant self-reinforcing directions in the Transformer value cache (implemented as low-rank damping). Across multiple large language models, RMR substantially reduces mode collapse and enables stable, high-quality generation at extremely low entropy rates (down to 0.8 nats/step), whereas standard decoding typically collapses near 2.0 nats/step.