Proximal Policy Optimization with Continuous Bounded Action Space via the Beta Distribution

TL;DR

This paper introduces a Beta distribution-based policy for continuous control in reinforcement learning, addressing the support mismatch of Gaussian policies and demonstrating superior performance and stability in benchmark tasks.

Contribution

It proposes using Beta distribution for policy modeling in PPO to better handle bounded action spaces, improving performance and training stability.

Findings

Beta policy outperforms Gaussian in expected reward

Beta policy shows more training stability

63% improvement in CarRacing success rate

Abstract

Reinforcement learning methods for continuous control tasks have evolved in recent years generating a family of policy gradient methods that rely primarily on a Gaussian distribution for modeling a stochastic policy. However, the Gaussian distribution has an infinite support, whereas real world applications usually have a bounded action space. This dissonance causes an estimation bias that can be eliminated if the Beta distribution is used for the policy instead, as it presents a finite support. In this work, we investigate how this Beta policy performs when it is trained by the Proximal Policy Optimization (PPO) algorithm on two continuous control tasks from OpenAI gym. For both tasks, the Beta policy is superior to the Gaussian policy in terms of agent's final expected reward, also showing more stability and faster convergence of the training process. For the CarRacing environment with high-dimensional image input, the agent's success rate was improved by 63% over the Gaussian policy.