Diversifying Policy Behaviors with Extrinsic Behavioral Curiosity

TL;DR

This paper introduces Extrinsic Behavioral Curiosity (EBC), a novel method that enhances the diversity and robustness of learned behaviors in imitation learning by incentivizing exploration of novel behaviors, significantly improving performance across various locomotion tasks.

Contribution

The paper proposes EBC, a new curiosity-driven approach that boosts behavioral diversity in imitation learning and QD-RL, outperforming existing methods and surpassing expert performance in some cases.

Findings

EBC improves QD-IRL performance by up to 185%.

EBC surpasses expert performance by 20% in Humanoid tasks.

EBC enhances gradient-based QD-RL algorithms.

Abstract

Imitation learning (IL) has shown promise in various applications (e.g. robot locomotion) but is often limited to learning a single expert policy, constraining behavior diversity and robustness in unpredictable real-world scenarios. To address this, we introduce Quality Diversity Inverse Reinforcement Learning (QD-IRL), a novel framework that integrates quality-diversity optimization with IRL methods, enabling agents to learn diverse behaviors from limited demonstrations. This work introduces Extrinsic Behavioral Curiosity (EBC), which allows agents to receive additional curiosity rewards from an external critic based on how novel the behaviors are with respect to a large behavioral archive. To validate the effectiveness of EBC in exploring diverse locomotion behaviors, we evaluate our method on multiple robot locomotion tasks. EBC improves the performance of QD-IRL instances with GAIL, VAIL, and DiffAIL across all included environments by up to 185%, 42%, and 150%, even surpassing expert performance by 20% in Humanoid. Furthermore, we demonstrate that EBC is applicable to Gradient-Arborescence-based Quality Diversity Reinforcement Learning (QD-RL) algorithms, where it substantially improves performance and provides a generic technique for learning behavioral-diverse policies. The source code of this work is provided at https://github.com/vanzll/EBC.