HIL: Hybrid Imitation Learning of Diverse Parkour Skills from Videos

TL;DR

This paper introduces a hybrid imitation learning framework that combines motion tracking and adversarial learning to enable simulated characters to perform diverse, natural parkour skills learned from videos, improving adaptability and skill diversity.

Contribution

The paper presents a novel hybrid imitation learning approach that integrates motion tracking with adversarial learning for diverse skill acquisition from videos.

Findings

Improved motion quality and skill diversity in simulated parkour tasks.

Enhanced adaptability to new environments compared to previous methods.

Achieved competitive task completion rates in challenging parkour scenarios.

Abstract

Recent data-driven methods leveraging deep reinforcement learning have been an effective paradigm for developing controllers that enable physically simulated characters to produce natural human-like behaviors. However, these data-driven methods often struggle to adapt to novel environments and compose diverse skills coherently to perform more complex tasks. To address these challenges, we propose a hybrid imitation learning (HIL) framework that combines motion tracking, for precise skill replication, with adversarial imitation learning, to enhance adaptability and skill composition. This hybrid learning framework is implemented through parallel multi-task environments and a unified observation space, featuring an agent-centric scene representation to facilitate effective learning from the hybrid parallel environments. Our framework trains a unified controller on parkour data sourced from Internet videos, enabling a simulated character to traverse through new environments using diverse and life-like parkour skills. Evaluations across challenging parkour environments demonstrate that our method improves motion quality, increases skill diversity, and achieves competitive task completion compared to previous learning-based methods.