Dealing with Sparse Rewards in Continuous Control Robotics via Heavy-Tailed Policies

TL;DR

This paper introduces HT-SPG, a novel heavy-tailed policy gradient algorithm that enhances exploration and learning efficiency in sparse reward continuous control tasks without needing expert demonstrations.

Contribution

We propose a heavy-tailed policy parametrization combined with a momentum-based policy gradient scheme to improve exploration and sample efficiency in sparse reward environments.

Findings

Consistent performance improvements across multiple benchmark tasks.

Faster convergence with fewer samples.

Enhanced exploration behavior without expert demonstrations.

Abstract

In this paper, we present a novel Heavy-Tailed Stochastic Policy Gradient (HT-PSG) algorithm to deal with the challenges of sparse rewards in continuous control problems. Sparse reward is common in continuous control robotics tasks such as manipulation and navigation, and makes the learning problem hard due to non-trivial estimation of value functions over the state space. This demands either reward shaping or expert demonstrations for the sparse reward environment. However, obtaining high-quality demonstrations is quite expensive and sometimes even impossible. We propose a heavy-tailed policy parametrization along with a modified momentum-based policy gradient tracking scheme (HT-SPG) to induce a stable exploratory behavior to the algorithm. The proposed algorithm does not require access to expert demonstrations. We test the performance of HT-SPG on various benchmark tasks of continuous control with sparse rewards such as 1D Mario, Pathological Mountain Car, Sparse Pendulum in OpenAI Gym, and Sparse MuJoCo environments (Hopper-v2). We show consistent performance improvement across all tasks in terms of high average cumulative reward. HT-SPG also demonstrates improved convergence speed with minimum samples, thereby emphasizing the sample efficiency of our proposed algorithm.