Imitation Learning from Suboptimal Demonstrations via Meta-Learning An Action Ranker

TL;DR

ILMAR is a novel imitation learning method that leverages both expert and suboptimal demonstrations through meta-learning an action ranker, significantly improving policy performance in scenarios with limited expert data.

Contribution

This paper introduces ILMAR, a meta-learning based approach that effectively utilizes suboptimal demonstrations by ranking and selectively integrating them into policy learning.

Findings

ILMAR outperforms previous methods on various tasks.

It effectively utilizes suboptimal demonstrations.

The approach improves policy performance with limited expert data.

Abstract

A major bottleneck in imitation learning is the requirement of a large number of expert demonstrations, which can be expensive or inaccessible. Learning from supplementary demonstrations without strict quality requirements has emerged as a powerful paradigm to address this challenge. However, previous methods often fail to fully utilize their potential by discarding non-expert data. Our key insight is that even demonstrations that fall outside the expert distribution but outperform the learned policy can enhance policy performance. To utilize this potential, we propose a novel approach named imitation learning via meta-learning an action ranker (ILMAR). ILMAR implements weighted behavior cloning (weighted BC) on a limited set of expert demonstrations along with supplementary demonstrations. It utilizes the functional of the advantage function to selectively integrate knowledge from the supplementary demonstrations. To make more effective use of supplementary demonstrations, we introduce meta-goal in ILMAR to optimize the functional of the advantage function by explicitly minimizing the distance between the current policy and the expert policy. Comprehensive experiments using extensive tasks demonstrate that ILMAR significantly outperforms previous methods in handling suboptimal demonstrations. Code is available at https://github.com/F-GOD6/ILMAR.