SLowRL: Safe Low-Rank Adaptation Reinforcement Learning for Locomotion

TL;DR

This paper introduces SLowRL, a safe and efficient reinforcement learning framework that combines low-rank adaptation and safety enforcement to improve sim-to-real transfer for robotic locomotion.

Contribution

It presents SLowRL, a novel method integrating Low-Rank Adaptation with safety recovery policies for safe, rapid real-world fine-tuning of locomotion policies.

Findings

Achieves 46.5% reduction in fine-tuning time.

Near-zero safety violations during real-world adaptation.

Rank-1 adaptation suffices for performance recovery.

Abstract

Sim-to-real transfer of locomotion policies often leads to performance degradation due to the inevitable sim-to-real gap. Naively fine-tuning these policies directly on hardware is problematic, as it poses risks of mechanical failure and suffers from high sample inefficiency. In this paper, we address the challenge of safely and efficiently fine-tuning reinforcement learning (RL) policies for dynamic locomotion tasks. Specifically, we focus on fine-tuning policies learned in simulation directly on hardware, while explicitly enforcing safety constraints. In doing so, we introduce SLowRL, a framework that combines Low-Rank Adaptation (LoRA) with training-time safety enforcement via a recovery policy. We evaluate our method both in simulation and on a real Unitree Go2 quadruped robot for jump and trot tasks. Experimental results show that our method achieves a $46.5\%$ reduction in fine-tuning time and near-zero safety violations compared to standard proximal policy optimization (PPO) baselines. Notably, we find that a rank-1 adaptation alone is sufficient to recover pre-trained performance in the real world, while maintaining stable and safe real-world fine-tuning. These results demonstrate the practicality of safe, efficient fine-tuning for dynamic real-world robotic applications.