Imitation Learning from Purified Demonstrations

TL;DR

This paper introduces a novel imitation learning method that purifies imperfect demonstrations using a diffusion process, improving learning effectiveness without requiring a high proportion of optimal data.

Contribution

We propose a diffusion-based purification technique for imperfect demonstrations, enabling more effective imitation learning from noisy data.

Findings

Effective noise reduction in demonstrations demonstrated on MuJoCo and RoboSuite

Theoretical bounds established for the distance between purified and optimal demonstrations

Improved imitation learning performance with purified data

Abstract

Imitation learning has emerged as a promising approach for addressing sequential decision-making problems, with the assumption that expert demonstrations are optimal. However, in real-world scenarios, most demonstrations are often imperfect, leading to challenges in the effectiveness of imitation learning. While existing research has focused on optimizing with imperfect demonstrations, the training typically requires a certain proportion of optimal demonstrations to guarantee performance. To tackle these problems, we propose to purify the potential noises in imperfect demonstrations first, and subsequently conduct imitation learning from these purified demonstrations. Motivated by the success of diffusion model, we introduce a two-step purification via diffusion process. In the first step, we apply a forward diffusion process to smooth potential noises in imperfect demonstrations by introducing additional noise. Subsequently, a reverse generative process is utilized to recover the optimal demonstration from the diffused ones. We provide theoretical evidence supporting our approach, demonstrating that the distance between the purified and optimal demonstration can be bounded. Empirical results on MuJoCo and RoboSuite demonstrate the effectiveness of our method from different aspects.