AlgaeDICE: Policy Gradient from Arbitrary Experience

TL;DR

AlgaeDICE introduces a new off-policy reinforcement learning method that enables policy gradient optimization from arbitrary experience without importance weighting, addressing data limitations in real-world applications.

Contribution

It reformulates max-return optimization to work with arbitrary off-policy data, eliminating the need for importance sampling in policy gradient computation.

Findings

The method matches on-policy policy gradients without importance weighting.

It demonstrates strong practical performance in RL tasks.

Theoretical properties ensure stable and efficient learning.

Abstract

In many real-world applications of reinforcement learning (RL), interactions with the environment are limited due to cost or feasibility. This presents a challenge to traditional RL algorithms since the max-return objective involves an expectation over on-policy samples. We introduce a new formulation of max-return optimization that allows the problem to be re-expressed by an expectation over an arbitrary behavior-agnostic and off-policy data distribution. We first derive this result by considering a regularized version of the dual max-return objective before extending our findings to unregularized objectives through the use of a Lagrangian formulation of the linear programming characterization of Q-values. We show that, if auxiliary dual variables of the objective are optimized, then the gradient of the off-policy objective is exactly the on-policy policy gradient, without any use of importance weighting. In addition to revealing the appealing theoretical properties of this approach, we also show that it delivers good practical performance.