DFPO: Scaling Value Modeling via Distributional Flow towards Robust and Generalizable LLM Post-Training

TL;DR

DFPO introduces a continuous flow-based distributional RL framework that enhances robustness and generalization in LLM post-training by modeling value functions as flows, stabilizing training under noisy supervision.

Contribution

It proposes a novel value flow modeling approach with risk and consistency controls, improving robustness and out-of-domain generalization in distributional RL for LLMs.

Findings

DFPO outperforms PPO and FlowRL on dialogue, math reasoning, and scientific tasks.

It achieves better training stability under noisy supervision.

DFPO enhances out-of-domain generalization in complex tasks.

Abstract

Training reinforcement learning (RL) systems in real-world environments remains challenging due to noisy supervision and poor out-of-domain (OOD) generalization, especially in LLM post-training. Recent distributional RL methods improve robustness by modeling values with multiple quantile points, but they still learn each quantile independently as a scalar. This results in rough-grained value representations that lack fine-grained conditioning on state information, struggling under complex and OOD conditions. We propose DFPO (Distributional Value Flow Policy Optimization with Conditional Risk and Consistency Control), a robust distributional RL framework that models values as continuous flows across time steps. By scaling value modeling through learning of a value flow field instead of isolated quantile predictions, DFPO captures richer state information for more accurate advantage estimation. To stabilize training under noisy feedback, DFPO further integrates conditional risk control and consistency constraints along value flow trajectories. Experiments on dialogue, math reasoning, and scientific tasks show that DFPO outperforms PPO, FlowRL, and other robust baselines under noisy supervision, achieving improved training stability and generalization.