Power Distribution Bridges Sampling, Self-Reward RL, and Self-Distillation

TL;DR

This paper reveals how power sampling, RL, and self-distillation are interconnected through the power distribution, enabling more efficient training and inference in large language models.

Contribution

It introduces power self-distillation, linking sampling, RL, and distillation via the power distribution, and demonstrates its effectiveness in reasoning tasks.

Findings

Power distribution connects sampling, RL, and distillation.

Power self-distillation can match power sampling performance with lower inference cost.

Power sampling increases self-reward, with true reward improvements depending on reward alignment.

Abstract

Recent analyses question whether reinforcement learning (RL) is responsible for strong reasoning in large language models (LLMs). At the same time, distillation and inference-time sampling, including power sampling, have emerged as effective ways to improve LLM performance. However, the relationship among RL, distillation, and sampling remains unclear. In this study, we focus on the power distribution, the target distribution of power sampling, and show that the power distribution bridges sampling, self-reward KL-regularized RL, and self-distillation. From the sampling perspective, we show that inexpensive local approximations cannot reproduce sequence-level power without information about possible suffixes. From the RL perspective, the power distribution is the closed-form optimizer of KL-regularized RL when the model's sequence-level log-probabilities are used as the reward. This identification leads to power self-distillation, an offline distillation surrogate that shares the same target distribution and amortizes the cost of power sampling into supervised training on teacher samples. We further show that power self-distillation can achieve self-reward sharpening, while improvement in a downstream true reward is governed by the covariance between true reward and self-reward under the power distribution. Experiments on reasoning tasks support our analysis: power sampling raises self-reward, true-reward gains depend on alignment with self-reward, and power self-distillation can match or exceed the performance of power sampling at much lower inference cost.