SD2AIL: Adversarial Imitation Learning from Synthetic Demonstrations via Diffusion Models

TL;DR

SD2AIL leverages diffusion models to generate synthetic demonstrations for adversarial imitation learning, enhancing performance and robustness with a prioritized replay strategy.

Contribution

It introduces a novel approach combining diffusion models with adversarial imitation learning and a prioritized demonstration replay mechanism.

Findings

Achieves an average return of 3441 on Hopper, surpassing state-of-the-art by 89.

Demonstrates effectiveness and robustness in simulation tasks.

Utilizes synthetic demonstrations to augment expert data.

Abstract

Adversarial Imitation Learning (AIL) is a dominant framework in imitation learning that infers rewards from expert demonstrations to guide policy optimization. Although providing more expert demonstrations typically leads to improved performance and greater stability, collecting such demonstrations can be challenging in certain scenarios. Inspired by the success of diffusion models in data generation, we propose SD2AIL, which utilizes synthetic demonstrations via diffusion models. We first employ a diffusion model in the discriminator to generate synthetic demonstrations as pseudo-expert data that augment the expert demonstrations. To selectively replay the most valuable demonstrations from the large pool of (pseudo-) expert demonstrations, we further introduce a prioritized expert demonstration replay strategy (PEDR). The experimental results on simulation tasks demonstrate the effectiveness and robustness of our method. In particular, in the Hopper task, our method achieves an average return of 3441, surpassing the state-of-the-art method by 89. Our code will be available at https://github.com/positron-lpc/SD2AIL.