Contraction Actor-Critic: Contraction Metric-Guided Reinforcement Learning for Robust Path Tracking

TL;DR

This paper introduces Contraction Actor-Critic (CAC), an RL algorithm that integrates control contraction metrics to learn robust, optimal path-tracking policies for systems with unknown dynamics, improving stability and performance.

Contribution

It proposes a novel RL framework that combines contraction metrics with actor-critic methods, enabling scalable, dynamics-informed control policy learning with stability guarantees.

Findings

CAC outperforms baseline methods in simulated environments.

CAC achieves robust path tracking in real-world robot experiments.

Theoretical analysis supports the stability and optimality of the approach.

Abstract

Control contraction metrics (CCMs) provide a framework to co-synthesize a controller and a corresponding contraction metric -- a positive-definite Riemannian metric under which a closed-loop system is guaranteed to be incrementally exponentially stable. However, the synthesized controller only ensures that all the trajectories of the system converge to one single trajectory and, as such, does not impose any notion of optimality across an entire trajectory. Furthermore, constructing CCMs requires a known dynamics model and non-trivial effort in solving an infinite-dimensional convex feasibility problem, which limits its scalability to complex systems featuring high dimensionality with uncertainty. To address these issues, we propose to integrate CCMs into reinforcement learning (RL), where CCMs provide dynamics-informed feedback for learning control policies that minimize cumulative tracking error under unknown dynamics. We show that our algorithm, called contraction actor-critic (CAC), formally enhances the capability of CCMs to provide a set of contracting policies with the long-term optimality of RL in a fully automated setting. Given a pre-trained dynamics model, CAC simultaneously learns a contraction metric generator (CMG) -- which generates a contraction metric -- and uses an actor-critic algorithm to learn an optimal tracking policy guided by that metric. We demonstrate the effectiveness of our algorithm relative to established baselines through extensive empirical studies, including simulated and real-world robot experiments, and provide a theoretical rationale for incorporating contraction theory into RL.