Learning Quadruped Locomotion Using Differentiable Simulation

TL;DR

This paper introduces a novel differentiable simulation framework that enables rapid learning of quadruped locomotion skills in simulation, outperforming reinforcement learning in efficiency and transferring successfully to real robots.

Contribution

The work presents a new differentiable simulation approach combining high-fidelity and surrogate models, facilitating fast, stable learning of complex quadruped movements in minutes.

Findings

Outperforms PPO in sample efficiency

Enables learning diverse gaits on challenging terrains

Achieves real-world quadruped locomotion

Abstract

This work explores the potential of using differentiable simulation for learning quadruped locomotion. Differentiable simulation promises fast convergence and stable training by computing low-variance first-order gradients using robot dynamics. However, its usage for legged robots is still limited to simulation. The main challenge lies in the complex optimization landscape of robotic tasks due to discontinuous dynamics. This work proposes a new differentiable simulation framework to overcome these challenges. Our approach combines a high-fidelity, non-differentiable simulator for forward dynamics with a simplified surrogate model for gradient backpropagation. This approach maintains simulation accuracy by aligning the robot states from the surrogate model with those of the precise, non-differentiable simulator. Our framework enables learning quadruped walking in simulation in minutes without parallelization. When augmented with GPU parallelization, our approach allows the quadruped robot to master diverse locomotion skills on challenging terrains in minutes. We demonstrate that differentiable simulation outperforms a reinforcement learning algorithm (PPO) by achieving significantly better sample efficiency while maintaining its effectiveness in handling large-scale environments. Our method represents one of the first successful applications of differentiable simulation to real-world quadruped locomotion, offering a compelling alternative to traditional RL methods.