Auxiliary-Hyperparameter-Free Sampling: Entropy Equilibrium for Text Generation

TL;DR

This paper introduces Entropy Equilibrium Sampling (EES), a hyperparameter-free method for text generation in large language models that balances entropy and probability to improve diversity, accuracy, and coherence without tuning.

Contribution

EES is a novel sampling approach that removes the need for hyperparameter tuning, simplifying deployment and enhancing performance in text generation tasks.

Findings

EES performs consistently well across different temperature settings.

EES achieves competitive accuracy and coherence in various tasks.

EES maintains diversity while improving overall text quality.

Abstract

Token sampling strategies critically influence text generation quality in large language models (LLMs). However, existing methods introduce additional hyperparameters, requiring extensive tuning and complicating deployment. We present Entropy Equilibrium Sampling (EES), an auxiliary hyperparameter-free approach inspired by information theory that can dynamically adjust candidate sets by balancing normalized entropy with probability mass. We evaluate EES on both reasoning and generation tasks across a range of model architectures. Our results show that EES consistently performs well across temperature settings, delivering competitive accuracy and coherence while maintaining diversity. By eliminating the need for hyperparameter tuning, EES greatly simplifies deployment while improving performance. Code is available at https://github.com/shuanncai/EES