Unlabeled Imperfect Demonstrations in Adversarial Imitation Learning

TL;DR

This paper introduces a positive-unlabeled adversarial imitation learning method that effectively learns from imperfect, unlabeled expert demonstrations, improving policy training in complex environments.

Contribution

It proposes a novel algorithm that handles unlabeled, imperfect demonstrations in adversarial imitation learning, with theoretical analysis and practical validation.

Findings

Effective learning from imperfect demonstrations demonstrated on MuJoCo and RoboSuite.

The method adapts dynamically to non-optimal expert data.

Theoretical analysis confirms self-paced learning from unlabeled data.

Abstract

Adversarial imitation learning has become a widely used imitation learning framework. The discriminator is often trained by taking expert demonstrations and policy trajectories as examples respectively from two categories (positive vs. negative) and the policy is then expected to produce trajectories that are indistinguishable from the expert demonstrations. But in the real world, the collected expert demonstrations are more likely to be imperfect, where only an unknown fraction of the demonstrations are optimal. Instead of treating imperfect expert demonstrations as absolutely positive or negative, we investigate unlabeled imperfect expert demonstrations as they are. A positive-unlabeled adversarial imitation learning algorithm is developed to dynamically sample expert demonstrations that can well match the trajectories from the constantly optimized agent policy. The trajectories of an initial agent policy could be closer to those non-optimal expert demonstrations, but within the framework of adversarial imitation learning, agent policy will be optimized to cheat the discriminator and produce trajectories that are similar to those optimal expert demonstrations. Theoretical analysis shows that our method learns from the imperfect demonstrations via a self-paced way. Experimental results on MuJoCo and RoboSuite platforms demonstrate the effectiveness of our method from different aspects.