PolicyFlow: Policy Optimization with Continuous Normalizing Flow in Reinforcement Learning

TL;DR

PolicyFlow introduces a novel reinforcement learning algorithm that employs continuous normalizing flow policies with a new importance ratio approximation, enhancing expressiveness and stability without full likelihood evaluation.

Contribution

It develops PolicyFlow, enabling the use of expressive flow-based policies in PPO without costly likelihood computations, and introduces the Brownian Regularizer for promoting policy diversity.

Findings

PolicyFlow outperforms PPO with Gaussian policies on multiple tasks.

It effectively captures multimodal action distributions.

The Brownian Regularizer improves policy diversity and stability.

Abstract

Among on-policy reinforcement learning algorithms, Proximal Policy Optimization (PPO) demonstrates is widely favored for its simplicity, numerical stability, and strong empirical performance. Standard PPO relies on surrogate objectives defined via importance ratios, which require evaluating policy likelihood that is typically straightforward when the policy is modeled as a Gaussian distribution. However, extending PPO to more expressive, high-capacity policy models such as continuous normalizing flows (CNFs), also known as flow-matching models, is challenging because likelihood evaluation along the full flow trajectory is computationally expensive and often numerically unstable. To resolve this issue, we propose PolicyFlow, a novel on-policy CNF-based reinforcement learning algorithm that integrates expressive CNF policies with PPO-style objectives without requiring likelihood evaluation along the full flow path. PolicyFlow approximates importance ratios using velocity field variations along a simple interpolation path, reducing computational overhead without compromising training stability. To further prevent mode collapse and further encourage diverse behaviors, we propose the Brownian Regularizer, an implicit policy entropy regularizer inspired by Brownian motion, which is conceptually elegant and computationally lightweight. Experiments on diverse tasks across various environments including MultiGoal, PointMaze, IsaacLab and MuJoCo Playground show that PolicyFlow achieves competitive or superior performance compared to PPO using Gaussian policies and flow-based baselines including FPO and DPPO. Notably, results on MultiGoal highlight PolicyFlow's ability to capture richer multimodal action distributions.