Min-$k$ Sampling: Decoupling Truncation from Temperature Scaling via Relative Logit Dynamics

TL;DR

Min-$k$ Sampling introduces a dynamic truncation method analyzing local logit distribution shapes to improve text generation quality and robustness across various tasks, independent of temperature sensitivity.

Contribution

It proposes Min-$k$ Sampling, a novel temperature-invariant truncation strategy that adapts to local confidence structures, outperforming existing methods.

Findings

Min-$k$ achieves strict temperature invariance.

It improves text quality across reasoning and creative tasks.

It maintains robustness even at extreme temperature settings.

Abstract

The quality of text generated by large language models depends critically on the decoding sampling strategy. While mainstream methods such as Top-$k$, Top-$p$, and Min-$p$ achieve a balance between diversity and accuracy through probability-space truncation, they share an inherent limitation: extreme sensitivity to the temperature parameter. Recent logit-space approaches like Top-$n\sigma$ achieve temperature invariance but rely on global statistics that are susceptible to long-tail noise, failing to capture fine-grained confidence structures among top candidates. We propose \textbf{Min-$k$ Sampling}, a novel dynamic truncation strategy that analyzes the local shape of the sorted logit distribution to identify "semantic cliffs": sharp transitions from high-confidence core tokens to uncertain long-tail tokens. By computing a position-weighted relative decay rate, Min-$k$ dynamically determines truncation boundaries at each generation step. We formally prove that Min-$k$ achieves strict temperature invariance and empirically demonstrate its low sensitivity to hyperparameter choices. Experiments on multiple reasoning benchmarks, creative writing tasks, and human evaluation show that Min-$k$ consistently improves text quality, maintaining robust performance even under extreme temperature settings where probability-based methods collapse. We make our code, models, and analysis tools publicly available.