Top-b: Entropic Regulation of Relative Probability Bands in Autoregressive Language Processes

TL;DR

This paper introduces Top-b, an adaptive decoding strategy for language models that dynamically adjusts candidate selection based on entropy, improving generation quality and consistency.

Contribution

It formalizes language generation as a trajectory on a probability manifold and proposes Top-b, a novel entropy-aware decoding method with theoretical and empirical validation.

Findings

Reduces generation entropy and variance.

Maintains reasoning accuracy on benchmarks.

Provides a theoretical framework for adaptive decoding.

Abstract

Probabilistic language generators are theoretically modeled as discrete stochastic processes, yet standard decoding strategies (Top-k, Top-p) impose static truncation rules that fail to accommodate the dynamic information density of natural language. This misalignment often forces a suboptimal trade-off: static bounds are either too restrictive for high-entropy creative generation or too permissive for low-entropy logical reasoning. In this work, we formalize the generation process as a trajectory through a relative probability manifold. We introduce Top-b (Adaptive Relative Band Sampling), a decoding strategy that regulates the candidate set via a dynamic bandwidth coefficient coupled strictly to the instantaneous Shannon entropy of the model's distribution. We provide a theoretical framework demonstrating that Top-b acts as a variance-minimizing operator on the tail distribution. Empirical validation on GPQA and GSM8K benchmarks indicates that Top-b significantly reduces generation entropy and inter-decoding variance while maintaining competitive reasoning accuracy, effectively approximating a self-regulating control system for autoregressive generation.