Text Generation Beyond Discrete Token Sampling

TL;DR

This paper introduces Mixture of Inputs (MoI), a training-free method that enhances autoregressive language models by maintaining richer token distribution information during generation, leading to improved performance on complex tasks.

Contribution

MoI is a novel, training-free approach that blends generated tokens with their distribution to preserve richer information during autoregressive text generation.

Findings

Improves text quality and reasoning in language models.

Enhances performance on mathematical reasoning, code, and PhD-level QA tasks.

Works across multiple large language models without additional training.

Abstract

In standard autoregressive generation, an LLM predicts the next-token distribution, samples a discrete token, and then discards the distribution, passing only the sampled token as new input. To preserve this distribution's rich information, we propose Mixture of Inputs (MoI), a training-free method for autoregressive generation. After generating a token following the standard paradigm, we construct a new input that blends the generated discrete token with the previously discarded token distribution. Specifically, we employ a Bayesian estimation method that treats the token distribution as the prior, the sampled token as the observation, and replaces the conventional one-hot vector with the continuous posterior expectation as the new model input. MoI allows the model to maintain a richer internal representation throughout the generation process, resulting in improved text quality and reasoning capabilities. On mathematical reasoning, code generation, and PhD-level QA tasks, MoI consistently improves performance across multiple models including QwQ-32B, Nemotron-Super-49B, Gemma-3-27B, and DAPO-Qwen-32B, with no additional training and negligible computational overhead.