Marginal Policy Gradients: A Unified Family of Estimators for Bounded Action Spaces with Applications

TL;DR

This paper introduces the angular policy gradient (APG), a new variance-reduced policy gradient method for directional control in reinforcement learning, providing a unified analysis with existing estimators and demonstrating improved performance in experiments.

Contribution

It presents the APG estimator for directional control, unifies variance reduction analysis for policy gradients, and extends variance reduction techniques to new action spaces.

Findings

APG significantly reduces variance compared to standard policy gradients.

Experimental results show APG outperforms existing methods in RTS game and navigation tasks.

Unified analysis offers stronger guarantees for variance reduction in policy gradient estimators.

Abstract

Many complex domains, such as robotics control and real-time strategy (RTS) games, require an agent to learn a continuous control. In the former, an agent learns a policy over $\mathbb{R}^d$ and in the latter, over a discrete set of actions each of which is parametrized by a continuous parameter. Such problems are naturally solved using policy based reinforcement learning (RL) methods, but unfortunately these often suffer from high variance leading to instability and slow convergence. Unnecessary variance is introduced whenever policies over bounded action spaces are modeled using distributions with unbounded support by applying a transformation $T$ to the sampled action before execution in the environment. Recently, the variance reduced clipped action policy gradient (CAPG) was introduced for actions in bounded intervals, but to date no variance reduced methods exist when the action is a direction, something often seen in RTS games. To this end we introduce the angular policy gradient (APG), a stochastic policy gradient method for directional control. With the marginal policy gradients family of estimators we present a unified analysis of the variance reduction properties of APG and CAPG; our results provide a stronger guarantee than existing analyses for CAPG. Experimental results on a popular RTS game and a navigation task show that the APG estimator offers a substantial improvement over the standard policy gradient.