Soft policy optimization using dual-track advantage estimator

TL;DR

This paper introduces a soft policy optimization method with a dual-track advantage estimator that balances exploration and exploitation, accelerating convergence and improving performance in reinforcement learning tasks.

Contribution

It proposes the dual-track advantage estimator (DTAE) that combines TD and GAE to enhance value function convergence in soft policy optimization.

Findings

Faster training speed compared to existing RL algorithms

Achieves state-of-the-art cumulative returns on Mujoco environments

Effectively balances exploration and exploitation through entropy adjustment

Abstract

In reinforcement learning (RL), we always expect the agent to explore as many states as possible in the initial stage of training and exploit the explored information in the subsequent stage to discover the most returnable trajectory. Based on this principle, in this paper, we soften the proximal policy optimization by introducing the entropy and dynamically setting the temperature coefficient to balance the opportunity of exploration and exploitation. While maximizing the expected reward, the agent will also seek other trajectories to avoid the local optimal policy. Nevertheless, the increase of randomness induced by entropy will reduce the train speed in the early stage. Integrating the temporal-difference (TD) method and the general advantage estimator (GAE), we propose the dual-track advantage estimator (DTAE) to accelerate the convergence of value functions and further enhance the performance of the algorithm. Compared with other on-policy RL algorithms on the Mujoco environment, the proposed method not only significantly speeds up the training but also achieves the most advanced results in cumulative return.