GPO: Growing Policy Optimization for Legged Robot Locomotion and Whole-Body Control

TL;DR

GPO is a novel reinforcement learning framework that progressively expands the action space during training, leading to more effective policy learning and better real-world performance in legged robot locomotion.

Contribution

GPO introduces a time-varying action transformation that preserves PPO stability and enhances exploration, improving policy training for torque-based legged robot control.

Findings

GPO-trained policies outperform baselines in simulation and hardware.

Zero-shot transfer of policies from simulation to real robots is successful.

GPO provides a general framework applicable across different legged robot platforms.

Abstract

Training reinforcement learning (RL) policies for legged robots remains challenging due to high-dimensional continuous actions, hardware constraints, and limited exploration. Existing methods for locomotion and whole-body control work well for position-based control with environment-specific heuristics (e.g., reward shaping, curriculum design, and manual initialization), but are less effective for torque-based control, where sufficiently exploring the action space and obtaining informative gradient signals for training is significantly more difficult. We introduce Growing Policy Optimization (GPO), a training framework that applies a time-varying action transformation to restrict the effective action space in the early stage, thereby encouraging more effective data collection and policy learning, and then progressively expands it to enhance exploration and achieve higher expected return. We prove that this transformation preserves the PPO update rule and introduces only bounded, vanishing gradient distortion, thereby ensuring stable training. We evaluate GPO on both quadruped and hexapod robots, including zero-shot deployment of simulation-trained policies on hardware. Policies trained with GPO consistently achieve better performance. These results suggest that GPO provides a general, environment-agnostic optimization framework for learning legged locomotion.