GoldenStart: Q-Guided Priors and Entropy Control for Distilling Flow Policies

TL;DR

GoldenStart (GSFlow) introduces Q-guided priors and entropy control to improve flow-matching policy distillation, enabling efficient, exploratory, and high-performance reinforcement learning policies with reduced inference latency.

Contribution

The paper proposes GSFlow, a novel policy distillation method using Q-guided priors and explicit entropy regulation to enhance exploration and inference efficiency in flow-based RL policies.

Findings

Outperforms prior state-of-the-art methods on continuous control benchmarks.

Effectively balances exploration and exploitation through entropy regularization.

Achieves faster inference with high-quality policy representations.

Abstract

Flow-matching policies hold great promise for reinforcement learning (RL) by capturing complex, multi-modal action distributions. However, their practical application is often hindered by prohibitive inference latency and ineffective online exploration. Although recent works have employed one-step distillation for fast inference, the structure of the initial noise distribution remains an overlooked factor that presents significant untapped potential. This overlooked factor, along with the challenge of controlling policy stochasticity, constitutes two critical areas for advancing distilled flow-matching policies. To overcome these limitations, we propose GoldenStart (GSFlow), a policy distillation method with Q-guided priors and explicit entropy control. Instead of initializing generation from uninformed noise, we introduce a Q-guided prior modeled by a conditional VAE. This state-conditioned prior repositions the starting points of the one-step generation process into high-Q regions, effectively providing a "golden start" that shortcuts the policy to promising actions. Furthermore, for effective online exploration, we enable our distilled actor to output a stochastic distribution instead of a deterministic point. This is governed by entropy regularization, allowing the policy to shift from pure exploitation to principled exploration. Our integrated framework demonstrates that by designing the generative startpoint and explicitly controlling policy entropy, it is possible to achieve efficient and exploratory policies, bridging the generative models and the practical actor-critic methods. We conduct extensive experiments on offline and online continuous control benchmarks, where our method significantly outperforms prior state-of-the-art approaches. Code will be available at https://github.com/ZhHe11/GSFlow-RL.