PowerFlow: Unlocking the Dual Nature of LLMs via Principled Distribution Matching

TL;DR

PowerFlow introduces a principled distribution matching framework for fine-tuning LLMs, enabling controlled sharpening or flattening of their output distributions to enhance reasoning or creativity, outperforming existing methods.

Contribution

It reformulates unsupervised LLM fine-tuning as a distribution matching problem using GFlowNet and a novel length-aware Trajectory-Balance objective, addressing biases and enabling dual control of LLM capabilities.

Findings

PowerFlow outperforms existing RLIF methods in various tasks.

It matches or exceeds supervised fine-tuning results.

The approach improves diversity and quality in creative tasks.

Abstract

Unsupervised Reinforcement Learning from Internal Feedback (RLIF) has emerged as a promising paradigm for eliciting the latent capabilities of Large Language Models (LLMs) without external supervision. However, current methods rely on heuristic intrinsic rewards, which often lack a well-defined theoretical optimization target and are prone to degenerative biases. In this work, we introduce PowerFlow, a principled framework that reformulates unsupervised fine-tuning as a distribution matching problem. By casting GFlowNet as an amortized variational sampler for unnormalized densities, we propose a length-aware Trajectory-Balance objective that explicitly neutralizes the structural length biases inherent in autoregressive generation. By targeting $\alpha$-power distributions, PowerFlow enables the directional elicitation of the dual nature of LLMs: sharpening the distribution ($\alpha > 1$) to intensify logical reasoning, or flattening it ($\alpha < 1$) to unlock expressive creativity. Extensive experiments demonstrate that PowerFlow consistently outperforms existing RLIF methods, matching or even exceeding supervised GRPO. Furthermore, by mitigating over-sharpening in aligned models, our approach achieves simultaneous gains in diversity and quality, shifting the Pareto frontier in creative tasks.