Imitating from auxiliary imperfect demonstrations via Adversarial Density Weighted Regression

TL;DR

This paper introduces Adversarial Density Regression (ADR), a novel one-step supervised imitation learning framework that effectively corrects policies trained on imperfect demonstrations to match expert distributions without relying on the Bellman operator.

Contribution

The paper presents ADR, a new IL method combining density-weighted behavioral cloning with theoretical guarantees, addressing limitations of previous algorithms like OOD issues and reliance on multi-step updates.

Findings

ADR outperforms existing IL algorithms on Gym-Mujoco tasks.

ADR achieves 89.5% improvement over IQL with ground truth rewards on Adroit and Kitchen tasks.

Theoretical analysis shows ADR effectively aligns policy distribution with expert distribution.

Abstract

We propose a novel one-step supervised imitation learning (IL) framework called Adversarial Density Regression (ADR). This IL framework aims to correct the policy learned on unknown-quality to match the expert distribution by utilizing demonstrations, without relying on the Bellman operator. Specifically, ADR addresses several limitations in previous IL algorithms: First, most IL algorithms are based on the Bellman operator, which inevitably suffer from cumulative offsets from sub-optimal rewards during multi-step update processes. Additionally, off-policy training frameworks suffer from Out-of-Distribution (OOD) state-actions. Second, while conservative terms help solve the OOD issue, balancing the conservative term is difficult. To address these limitations, we fully integrate a one-step density-weighted Behavioral Cloning (BC) objective for IL with auxiliary imperfect demonstration. Theoretically, we demonstrate that this adaptation can effectively correct the distribution of policies trained on unknown-quality datasets to align with the expert policy's distribution. Moreover, the difference between the empirical and the optimal value function is proportional to the upper bound of ADR's objective, indicating that minimizing ADR's objective is akin to approaching the optimal value. Experimentally, we validated the performance of ADR by conducting extensive evaluations. Specifically, ADR outperforms all of the selected IL algorithms on tasks from the Gym-Mujoco domain. Meanwhile, it achieves an 89.5% improvement over IQL when utilizing ground truth rewards on tasks from the Adroit and Kitchen domains. Our codebase will be released at: https://github.com/stevezhangzA/Adverserial_Density_Regression.