SRL-VIC: A Variable Stiffness-Based Safe Reinforcement Learning for Contact-Rich Robotic Tasks

TL;DR

SRL-VIC introduces a safe reinforcement learning framework with variable stiffness control, enabling contact-rich robotic tasks to be performed safely and efficiently, with successful simulation-to-real transfer demonstrated.

Contribution

It presents a novel model-free safe RL method combining a variable impedance controller and pre-trained safety modules for contact-rich tasks.

Findings

Outperforms baselines in contact-rich maze tasks

Achieves a good balance between safety and task efficiency

Successfully transfers from simulation to real robot without fine-tuning

Abstract

Reinforcement learning (RL) has emerged as a promising paradigm in complex and continuous robotic tasks, however, safe exploration has been one of the main challenges, especially in contact-rich manipulation tasks in unstructured environments. Focusing on this issue, we propose SRL-VIC: a model-free safe RL framework combined with a variable impedance controller (VIC). Specifically, safety critic and recovery policy networks are pre-trained where safety critic evaluates the safety of the next action using a risk value before it is executed and the recovery policy suggests a corrective action if the risk value is high. Furthermore, the policies are updated online where the task policy not only achieves the task but also modulates the stiffness parameters to keep a safe and compliant profile. A set of experiments in contact-rich maze tasks demonstrate that our framework outperforms the baselines (without the recovery mechanism and without the VIC), yielding a good trade-off between efficient task accomplishment and safety guarantee. We show our policy trained on simulation can be deployed on a physical robot without fine-tuning, achieving successful task completion with robustness and generalization. The video is available at https://youtu.be/ksWXR3vByoQ.