Variance Reduction based Partial Trajectory Reuse to Accelerate Policy Gradient Optimization

TL;DR

This paper introduces a variance reduction technique that reuses partial trajectories through importance sampling to improve policy gradient optimization in infinite-horizon MDPs, enabling faster convergence and better performance.

Contribution

It proposes a novel variance reduction experience replay method that selectively reuses relevant partial trajectories using mixture likelihood ratios, enhancing policy learning efficiency.

Findings

Improved convergence speed of policy optimization algorithms.

Enhanced performance of actor-critic and proximal policy optimization methods.

Effective reuse of partial trajectories in low-data and online settings.

Abstract

Built on our previous study on green simulation assisted policy gradient (GS-PG) focusing on trajectory-based reuse, in this paper, we consider infinite-horizon Markov Decision Processes and create a new importance sampling based policy gradient optimization approach to support dynamic decision making. The existing GS-PG method was designed to learn from complete episodes or process trajectories, which limits its applicability to low-data situations and flexible online process control. To overcome this limitation, the proposed approach can selectively reuse the most related partial trajectories, i.e., the reuse unit is based on per-step or per-decision historical observations. In specific, we create a mixture likelihood ratio (MLR) based policy gradient optimization that can leverage the information from historical state-action transitions generated under different behavioral policies. The proposed variance reduction experience replay (VRER) approach can intelligently select and reuse most relevant transition observations, improve the policy gradient estimation, and accelerate the learning of optimal policy. Our empirical study demonstrates that it can improve optimization convergence and enhance the performance of state-of-the-art policy optimization approaches such as actor-critic method and proximal policy optimizations.