Constrained Skill Discovery: Quadruped Locomotion with Unsupervised Reinforcement Learning

TL;DR

This paper introduces a new unsupervised reinforcement learning method for quadruped robots that enhances skill diversity and stability, enabling accurate zero-shot navigation without task-specific rewards.

Contribution

The authors propose a novel constrained skill discovery approach using a norm-matching objective, improving state coverage and locomotion stability over previous methods.

Findings

Richer state space coverage achieved

More stable and controllable locomotion behaviors learned

Successful zero-shot navigation on real robot

Abstract

Representation learning and unsupervised skill discovery can allow robots to acquire diverse and reusable behaviors without the need for task-specific rewards. In this work, we use unsupervised reinforcement learning to learn a latent representation by maximizing the mutual information between skills and states subject to a distance constraint. Our method improves upon prior constrained skill discovery methods by replacing the latent transition maximization with a norm-matching objective. This not only results in a much a richer state space coverage compared to baseline methods, but allows the robot to learn more stable and easily controllable locomotive behaviors. We successfully deploy the learned policy on a real ANYmal quadruped robot and demonstrate that the robot can accurately reach arbitrary points of the Cartesian state space in a zero-shot manner, using only an intrinsic skill discovery and standard regularization rewards.