Decoding in Geometry: Alleviating Embedding-Space Crowding for Complex Reasoning

TL;DR

This paper identifies embedding-space crowding as a key factor affecting reasoning in large language models and introduces CraEG, a geometry-aware sampling method that improves reasoning performance without additional training.

Contribution

It uncovers the phenomenon of embedding-space crowding and proposes CraEG, a novel, training-free sampling technique that mitigates crowding to enhance reasoning in LLMs.

Findings

CraEG improves reasoning accuracy on benchmarks.

Embedding-space crowding correlates with reasoning success.

CraEG enhances diversity and robustness of generated outputs.

Abstract

Sampling-based decoding underlies complex reasoning in large language models (LLMs), where decoding strategies critically shape model behavior. Temperature- and truncation-based methods reshape the next-token distribution through global probability reweighting or thresholding to balance the quality-diversity tradeoff. However, they operate solely on token probabilities, ignoring fine-grained relationships among tokens in the embedding space. We uncover a novel phenomenon, embedding-space crowding, where the next-token distribution concentrates its probability mass on geometrically close tokens in the embedding space. We quantify crowding at multiple granularities and find a statistical association with reasoning success in mathematical problem solving. Motivated by this finding, we propose CraEG, a plug-and-play sampling method that mitigates crowding through geometry-guided reweighting. CraEG is training-free, single-pass, and compatible with standard sampling strategies. Experiments on multiple models and benchmarks demonstrate improved generation performance, with gains in robustness and diversity metrics.