ContractionPPO: Certified Reinforcement Learning via Differentiable Contraction Layers

TL;DR

ContractionPPO integrates a neural contraction metric into reinforcement learning to certify and enhance the robustness and stability of legged robot control policies, ensuring reliable performance in unstructured environments.

Contribution

It introduces a novel method combining RL with a differentiable contraction metric layer for certifiable stability in legged robot control.

Findings

Demonstrates robust quadruped locomotion under external perturbations.

Provides theoretical guarantees of incremental exponential stability.

Shows successful transfer from simulation to real-world deployment.

Abstract

Legged locomotion in unstructured environments demands not only high-performance control policies but also formal guarantees to ensure robustness under perturbations. Control methods often require carefully designed reference trajectories, which are challenging to construct in high-dimensional, contact-rich systems such as quadruped robots. In contrast, Reinforcement Learning (RL) directly learns policies that implicitly generate motion, and uniquely benefits from access to privileged information, such as full state and dynamics during training, that is not available at deployment. We present ContractionPPO, a framework for certified robust planning and control of legged robots by augmenting Proximal Policy Optimization (PPO) RL with a state-dependent contraction metric layer. This approach enables the policy to maximize performance while simultaneously producing a contraction metric that certifies incremental exponential stability of the simulated closed-loop system. The metric is parameterized as a Lipschitz neural network and trained jointly with the policy, either in parallel or as an auxiliary head of the PPO backbone. While the contraction metric is not deployed during real-world execution, we derive upper bounds on the worst-case contraction rate and show that these bounds ensure the learned contraction metric generalizes from simulation to real-world deployment. Our hardware experiments on quadruped locomotion demonstrate that ContractionPPO enables robust, certifiably stable control even under strong external perturbations.