Diverse Skill Discovery for Quadruped Robots via Unsupervised Learning

TL;DR

This paper introduces an unsupervised skill discovery method for quadruped robots that enhances diversity and efficiency of learned behaviors by preventing skill overlap and reward hacking through novel architecture and multi-discriminator design.

Contribution

The paper proposes the Orthogonal Mixture-of-Experts architecture and a multi-discriminator framework to improve skill diversity and training efficiency in unsupervised learning for quadruped robots.

Findings

Achieved an 18.3% increase in state-space coverage.

Demonstrated diverse locomotion skills on a 12-DOF quadruped.

Enhanced training efficiency compared to baseline methods.

Abstract

Reinforcement learning necessitates meticulous reward shaping by specialists to elicit target behaviors, while imitation learning relies on costly task-specific data. In contrast, unsupervised skill discovery can potentially reduce these burdens by learning a diverse repertoire of useful skills driven by intrinsic motivation. However, existing methods exhibit two key limitations: they typically rely on a single policy to master a versatile repertoire of behaviors without modeling the shared structure or distinctions among them, which results in low learning efficiency; moreover, they are susceptible to reward hacking, where the reward signal increases and converges rapidly while the learned skills display insufficient actual diversity. In this work, we introduce an Orthogonal Mixture-of-Experts (OMoE) architecture that prevents diverse behaviors from collapsing into overlapping representations, enabling a single policy to master a wide spectrum of locomotion skills. In addition, we design a multi-discriminator framework in which different discriminators operate on distinct observation spaces, effectively mitigating reward hacking. We evaluated our method on the 12-DOF Unitree A1 quadruped robot, demonstrating a diverse set of locomotion skills. Our experiments demonstrate that the proposed framework boosts training efficiency and yields an 18.3\% expansion in state-space coverage compared to the baseline.