Diffusion Policies with Value-Conditional Optimization for Offline Reinforcement Learning

TL;DR

DIVO introduces a value-conditional diffusion approach for offline reinforcement learning, balancing conservatism and exploration by selectively generating and filtering actions based on advantage values, leading to improved performance on benchmarks.

Contribution

The paper proposes DIVO, a novel diffusion-based offline RL method that uses advantage-guided training and filtering to enhance policy quality and dataset coverage.

Findings

DIVO outperforms state-of-the-art methods on D4RL benchmarks.

DIVO achieves significant improvements in average returns on locomotion tasks.

DIVO excels in the challenging AntMaze domain with sparse rewards.

Abstract

In offline reinforcement learning, value overestimation caused by out-of-distribution (OOD) actions significantly limits policy performance. Recently, diffusion models have been leveraged for their strong distribution-matching capabilities, enforcing conservatism through behavior policy constraints. However, existing methods often apply indiscriminate regularization to redundant actions in low-quality datasets, resulting in excessive conservatism and an imbalance between the expressiveness and efficiency of diffusion modeling. To address these issues, we propose DIffusion policies with Value-conditional Optimization (DIVO), a novel approach that leverages diffusion models to generate high-quality, broadly covered in-distribution state-action samples while facilitating efficient policy improvement. Specifically, DIVO introduces a binary-weighted mechanism that utilizes the advantage values of actions in the offline dataset to guide diffusion model training. This enables a more precise alignment with the dataset's distribution while selectively expanding the boundaries of high-advantage actions. During policy improvement, DIVO dynamically filters high-return-potential actions from the diffusion model, effectively guiding the learned policy toward better performance. This approach achieves a critical balance between conservatism and explorability in offline RL. We evaluate DIVO on the D4RL benchmark and compare it against state-of-the-art baselines. Empirical results demonstrate that DIVO achieves superior performance, delivering significant improvements in average returns across locomotion tasks and outperforming existing methods in the challenging AntMaze domain, where sparse rewards pose a major difficulty.