Revisiting Generative Policies: A Simpler Reinforcement Learning Algorithmic Perspective

TL;DR

This paper analyzes and simplifies generative policy training methods in reinforcement learning, proposing two approaches that achieve state-of-the-art results and providing a unified framework for their deployment.

Contribution

It introduces GMPO and GMPG as simpler, effective training objectives for generative policies, along with a standardized experimental framework called GenerativeRL.

Findings

GMPO employs advantage-weighted regression for simplicity.

GMPG provides a stable policy gradient implementation.

Proposed methods outperform previous approaches on offline RL datasets.

Abstract

Generative models, particularly diffusion models, have achieved remarkable success in density estimation for multimodal data, drawing significant interest from the reinforcement learning (RL) community, especially in policy modeling in continuous action spaces. However, existing works exhibit significant variations in training schemes and RL optimization objectives, and some methods are only applicable to diffusion models. In this study, we compare and analyze various generative policy training and deployment techniques, identifying and validating effective designs for generative policy algorithms. Specifically, we revisit existing training objectives and classify them into two categories, each linked to a simpler approach. The first approach, Generative Model Policy Optimization (GMPO), employs a native advantage-weighted regression formulation as the training objective, which is significantly simpler than previous methods. The second approach, Generative Model Policy Gradient (GMPG), offers a numerically stable implementation of the native policy gradient method. We introduce a standardized experimental framework named GenerativeRL. Our experiments demonstrate that the proposed methods achieve state-of-the-art performance on various offline-RL datasets, offering a unified and practical guideline for training and deploying generative policies.