Robust Offline Imitation Learning from Diverse Auxiliary Data

TL;DR

ROIDA is a robust offline imitation learning method that effectively utilizes diverse auxiliary data by identifying high-quality transitions and applying temporal difference learning, outperforming prior approaches without relying on strict data assumptions.

Contribution

ROIDA introduces a novel approach combining reward-based filtering and temporal difference learning to leverage diverse auxiliary data in offline imitation learning.

Findings

ROIDA outperforms prior methods across various auxiliary datasets.

It maintains robust performance without assumptions on data quality.

The method effectively leverages unlabeled auxiliary data.

Abstract

Offline imitation learning enables learning a policy solely from a set of expert demonstrations, without any environment interaction. To alleviate the issue of distribution shift arising due to the small amount of expert data, recent works incorporate large numbers of auxiliary demonstrations alongside the expert data. However, the performance of these approaches rely on assumptions about the quality and composition of the auxiliary data, and they are rarely successful when those assumptions do not hold. To address this limitation, we propose Robust Offline Imitation from Diverse Auxiliary Data (ROIDA). ROIDA first identifies high-quality transitions from the entire auxiliary dataset using a learned reward function. These high-reward samples are combined with the expert demonstrations for weighted behavioral cloning. For lower-quality samples, ROIDA applies temporal difference learning to steer the policy towards high-reward states, improving long-term returns. This two-pronged approach enables our framework to effectively leverage both high and low-quality data without any assumptions. Extensive experiments validate that ROIDA achieves robust and consistent performance across multiple auxiliary datasets with diverse ratios of expert and non-expert demonstrations. ROIDA effectively leverages unlabeled auxiliary data, outperforming prior methods reliant on specific data assumptions. Our code is available at https://github.com/uditaghosh/roida.