Positive-Only Drifting Policy Optimization

TL;DR

This paper introduces PODPO, a novel likelihood-free, gradient-clipping-free generative method for online reinforcement learning that updates policies using only positive samples to improve high-return actions.

Contribution

PODPO is a new generative policy optimization approach that relies solely on positive samples and avoids gradient clipping, enhancing online RL performance.

Findings

PODPO effectively steers policies toward high-return regions.

It exploits local smoothness for proactive error prevention.

The method operates without gradient clipping or post-hoc penalization.

Abstract

In the field of online reinforcement learning (RL), traditional Gaussian policies and flow-based methods are often constrained by their unimodal expressiveness, complex gradient clipping, or stringent trust-region requirements. Moreover, they all rely on post-hoc penalization of negative samples to correct erroneous actions. This paper introduces Positive-Only Drifting Policy Optimization (PODPO), a likelihood-free and gradient-clipping-free generative approach for online RL. By leveraging the drifting model, PODPO performs policy updates via advantage-weighted local contrastive drifting. Relying solely on positive-advantage samples, it elegantly steers actions toward high-return regions while exploiting the inherent local smoothness of the generative model to enable proactive error prevention. In doing so, PODPO opens a promising new pathway for generative policy learning in online settings.