A Real-World Quadrupedal Locomotion Benchmark for Offline Reinforcement Learning

TL;DR

This paper introduces a new benchmark dataset for offline reinforcement learning in quadrupedal locomotion, enabling evaluation of algorithms on realistic, MPC-collected data to advance stable and agile legged robot control.

Contribution

It provides the first comprehensive benchmark dataset for offline RL in quadruped locomotion, collected via MPC, and evaluates 11 algorithms, highlighting current strengths and gaps.

Findings

Best algorithms outperform some RL methods

Algorithms still lag behind MPC in stability and adaptation

Benchmark facilitates future offline RL research in legged robots

Abstract

Online reinforcement learning (RL) methods are often data-inefficient or unreliable, making them difficult to train on real robotic hardware, especially quadruped robots. Learning robotic tasks from pre-collected data is a promising direction. Meanwhile, agile and stable legged robotic locomotion remains an open question in their general form. Offline reinforcement learning (ORL) has the potential to make breakthroughs in this challenging field, but its current bottleneck lies in the lack of diverse datasets for challenging realistic tasks. To facilitate the development of ORL, we benchmarked 11 ORL algorithms in the realistic quadrupedal locomotion dataset. Such dataset is collected by the classic model predictive control (MPC) method, rather than the model-free online RL method commonly used by previous benchmarks. Extensive experimental results show that the best-performing ORL algorithms can achieve competitive performance compared with the model-free RL, and even surpass it in some tasks. However, there is still a gap between the learning-based methods and MPC, especially in terms of stability and rapid adaptation. Our proposed benchmark will serve as a development platform for testing and evaluating the performance of ORL algorithms in real-world legged locomotion tasks.