Learning to Discern: Imitating Heterogeneous Human Demonstrations with Preference and Representation Learning

TL;DR

This paper introduces Learning to Discern (L2D), an offline imitation learning framework that uses preference and representation learning to evaluate and learn from heterogeneous human demonstrations of varying quality, improving policy performance.

Contribution

L2D is a novel framework that learns a latent representation and quality evaluator from limited labeled demonstrations, handling diverse styles and suboptimal data in imitation learning.

Findings

L2D effectively assesses demonstration quality across diverse styles.

L2D improves policy performance in simulation and real robot tasks.

The approach generalizes well to new demonstrators and styles.

Abstract

Practical Imitation Learning (IL) systems rely on large human demonstration datasets for successful policy learning. However, challenges lie in maintaining the quality of collected data and addressing the suboptimal nature of some demonstrations, which can compromise the overall dataset quality and hence the learning outcome. Furthermore, the intrinsic heterogeneity in human behavior can produce equally successful but disparate demonstrations, further exacerbating the challenge of discerning demonstration quality. To address these challenges, this paper introduces Learning to Discern (L2D), an offline imitation learning framework for learning from demonstrations with diverse quality and style. Given a small batch of demonstrations with sparse quality labels, we learn a latent representation for temporally embedded trajectory segments. Preference learning in this latent space trains a quality evaluator that generalizes to new demonstrators exhibiting different styles. Empirically, we show that L2D can effectively assess and learn from varying demonstrations, thereby leading to improved policy performance across a range of tasks in both simulations and on a physical robot.