Contraction Metric Based Safe Reinforcement Learning Force Control for a Hydraulic Actuator with Real-World Training

TL;DR

This paper presents a safe reinforcement learning approach for hydraulic force control using contraction metric certificates, enabling real-world training and improved safety in complex, nonlinear hydraulic systems.

Contribution

It introduces a contraction metric-based safety filter and a data-driven simulation pretraining method for RL in hydraulic force control, validated on real hardware.

Findings

Real-world training enhances force-tracking performance.

Contraction filter reduces chattering and instability.

Method outperforms simulation-trained and fixed-gain controllers.

Abstract

Force control in hydraulic actuators is notoriously difficult due to strong nonlinearities, uncertainties, and the high risks associated with unsafe exploration during learning. This paper investigates safe reinforcement learning (RL) for hy draulic force control with real-world training using contraction metric certificates. A data-driven model of a hydraulic actuator, identified from experimental data, is employed for simulation based pretraining of a Soft Actor-Critic (SAC) policy that adapts the PI gains of a feedback-linearization (FL) controller. To reduce instability during online training, we propose a quadratic-programming (QP) contraction filter that leverages a learned contraction metric to enforce approximate exponential convergence of trajectories, applying minimal corrections to the policy output. The approach is validated on a hydraulic test bench, where the RL controller is trained directly on hardware and benchmarked against a simulation-trained agent and a fixed-gain baseline. Experimental results show that real-hardware training improves force-tracking performance compared to both alternatives, while the contraction filter mitigates chattering and instabilities. These findings suggest that contraction-based certificates can enable safe RL in high force hydraulic systems, though robustness at extreme operating conditions remains a challenge.