CTS: Concurrent Teacher-Student Reinforcement Learning for Legged Locomotion

TL;DR

This paper introduces CTS, a concurrent teacher-student reinforcement learning framework for legged robots that trains both policies simultaneously, leading to improved robustness and agility in uneven terrain locomotion.

Contribution

The paper presents a novel concurrent training architecture and a modified PPO scheme for legged locomotion, outperforming traditional sequential teacher-student methods.

Findings

Reduces velocity tracking error by up to 20% in simulations.

Demonstrates robust and agile locomotion in indoor and outdoor experiments.

Outperforms state-of-the-art approaches in quantitative comparisons.

Abstract

Thanks to recent explosive developments of data-driven learning methodologies, reinforcement learning (RL) emerges as a promising solution to address the legged locomotion problem in robotics. In this paper, we propose CTS, a novel Concurrent Teacher-Student reinforcement learning architecture for legged locomotion over uneven terrains. Different from conventional teacher-student architecture that trains the teacher policy via RL first and then transfers the knowledge to the student policy through supervised learning, our proposed architecture trains teacher and student policy networks concurrently under the reinforcement learning paradigm. To this end, we develop a new training scheme based on a modified proximal policy gradient (PPO) method that exploits data samples collected from the interactions between both the teacher and the student policies with the environment. The effectiveness of the proposed architecture and the new training scheme is demonstrated through substantial quantitative simulation comparisons with the state-of-the-art approaches and extensive indoor and outdoor experiments with quadrupedal and point-foot bipedal robot platforms, showcasing robust and agile locomotion capability. Quantitative simulation comparisons show that our approach reduces the average velocity tracking error by up to 20% compared to the two-stage teacher-student, demonstrating significant superiority in addressing blind locomotion tasks. Videos are available at https://clearlab-sustech.github.io/concurrentTS.