PPO-UE: Proximal Policy Optimization via Uncertainty-Aware Exploration

TL;DR

This paper introduces PPO-UE, an enhanced version of PPO that incorporates self-adaptive uncertainty-aware exploration to improve training stability, convergence speed, and performance in continuous control tasks.

Contribution

PPO-UE is a novel PPO variant that uses ratio uncertainty levels for adaptive exploration, addressing stability issues and boosting performance.

Findings

PPO-UE outperforms baseline PPO in Roboschool tasks.

Sensitivity analysis shows optimal ratio uncertainty levels improve results.

PPO-UE enhances convergence speed and stability.

Abstract

Proximal Policy Optimization (PPO) is a highly popular policy-based deep reinforcement learning (DRL) approach. However, we observe that the homogeneous exploration process in PPO could cause an unexpected stability issue in the training phase. To address this issue, we propose PPO-UE, a PPO variant equipped with self-adaptive uncertainty-aware explorations (UEs) based on a ratio uncertainty level. The proposed PPO-UE is designed to improve convergence speed and performance with an optimized ratio uncertainty level. Through extensive sensitivity analysis by varying the ratio uncertainty level, our proposed PPO-UE considerably outperforms the baseline PPO in Roboschool continuous control tasks.