Off-OAB: Off-Policy Policy Gradient Method with Optimal Action-Dependent Baseline

TL;DR

This paper introduces Off-OAB, an off-policy policy gradient method with an optimal action-dependent baseline that reduces variance and improves sample efficiency in reinforcement learning, outperforming existing methods on benchmark tasks.

Contribution

The paper proposes a novel off-policy policy gradient method with an optimal action-dependent baseline that minimizes variance while maintaining unbiasedness, with practical approximations for efficiency.

Findings

Outperforms state-of-the-art methods on most OpenAI Gym and MuJoCo tasks.

Effectively reduces variance of the off-policy policy gradient estimator.

Enhances sample efficiency in reinforcement learning training.

Abstract

Policy-based methods have achieved remarkable success in solving challenging reinforcement learning problems. Among these methods, off-policy policy gradient methods are particularly important due to that they can benefit from off-policy data. However, these methods suffer from the high variance of the off-policy policy gradient (OPPG) estimator, which results in poor sample efficiency during training. In this paper, we propose an off-policy policy gradient method with the optimal action-dependent baseline (Off-OAB) to mitigate this variance issue. Specifically, this baseline maintains the OPPG estimator's unbiasedness while theoretically minimizing its variance. To enhance practical computational efficiency, we design an approximated version of this optimal baseline. Utilizing this approximation, our method (Off-OAB) aims to decrease the OPPG estimator's variance during policy optimization. We evaluate the proposed Off-OAB method on six representative tasks from OpenAI Gym and MuJoCo, where it demonstrably surpasses state-of-the-art methods on the majority of these tasks.