Partial advantage estimator for proximal policy optimization

TL;DR

This paper introduces a partial advantage estimator for proximal policy optimization that reduces bias caused by truncated trajectories, leading to improved empirical performance in reinforcement learning tasks.

Contribution

It proposes a novel partial GAE method that mitigates bias from incomplete trajectories, enhancing policy gradient estimation accuracy.

Findings

Partial GAE reduces bias in advantage estimation.

The method improves performance in MuJoCo and μRTS environments.

Experimental results show better empirical outcomes with partial GAE.

Abstract

Estimation of value in policy gradient methods is a fundamental problem. Generalized Advantage Estimation (GAE) is an exponentially-weighted estimator of an advantage function similar to $\lambda$-return. It substantially reduces the variance of policy gradient estimates at the expense of bias. In practical applications, a truncated GAE is used due to the incompleteness of the trajectory, which results in a large bias during estimation. To address this challenge, instead of using the entire truncated GAE, we propose to take a part of it when calculating updates, which significantly reduces the bias resulting from the incomplete trajectory. We perform experiments in MuJoCo and $\mu$RTS to investigate the effect of different partial coefficient and sampling lengths. We show that our partial GAE approach yields better empirical results in both environments.