AM-PPO: (Advantage) Alpha-Modulation with Proximal Policy Optimization

TL;DR

AM-PPO introduces an adaptive advantage modulation mechanism to improve the stability and performance of PPO in reinforcement learning by dynamically scaling advantage estimates based on their statistical properties.

Contribution

The paper proposes a novel advantage modulation technique with an alpha controller for PPO, enhancing stability and learning efficiency in reinforcement learning.

Findings

Achieves superior reward trajectories on continuous control benchmarks.

Reduces the need for clipping in adaptive optimizers.

Demonstrates improved stability and sustained learning progression.

Abstract

Proximal Policy Optimization (PPO) is a widely used reinforcement learning algorithm that heavily relies on accurate advantage estimates for stable and efficient training. However, raw advantage signals can exhibit significant variance, noise, and scale-related issues, impeding optimal learning performance. To address this challenge, we introduce Advantage Modulation PPO (AM-PPO), a novel enhancement of PPO that adaptively modulates advantage estimates using a dynamic, non-linear scaling mechanism. This adaptive modulation employs an alpha controller that dynamically adjusts the scaling factor based on evolving statistical properties of the advantage signals, such as their norm, variance, and a predefined target saturation level. By incorporating a tanh-based gating function driven by these adaptively scaled advantages, AM-PPO reshapes the advantage signals to stabilize gradient updates and improve the conditioning of the policy gradient landscape. Crucially, this modulation also influences value function training by providing consistent and adaptively conditioned learning targets. Empirical evaluations across standard continuous control benchmarks demonstrate that AM-PPO achieves superior reward trajectories, exhibits sustained learning progression, and significantly reduces the clipping required by adaptive optimizers. These findings underscore the potential of advantage modulation as a broadly applicable technique for enhancing reinforcement learning optimization.