Provable Guarantees for Generative Behavior Cloning: Bridging Low-Level Stability and High-Level Behavior

TL;DR

This paper introduces a theoretical framework for behavior cloning using generative models, emphasizing low-level stability and a novel noise augmentation technique to closely imitate expert trajectories in complex tasks.

Contribution

It provides a new theoretical analysis linking low-level controller stability with high-level imitation quality, and proposes a noise augmentation method to ensure trajectory distribution closeness.

Findings

The framework guarantees imitation quality under stability assumptions.

Adding augmentation noise improves imitation accuracy with minimal degradation.

Empirical validation supports the proposed theoretical insights.

Abstract

We propose a theoretical framework for studying behavior cloning of complex expert demonstrations using generative modeling. Our framework invokes low-level controllers - either learned or implicit in position-command control - to stabilize imitation around expert demonstrations. We show that with (a) a suitable low-level stability guarantee and (b) a powerful enough generative model as our imitation learner, pure supervised behavior cloning can generate trajectories matching the per-time step distribution of essentially arbitrary expert trajectories in an optimal transport cost. Our analysis relies on a stochastic continuity property of the learned policy we call "total variation continuity" (TVC). We then show that TVC can be ensured with minimal degradation of accuracy by combining a popular data-augmentation regimen with a novel algorithmic trick: adding augmentation noise at execution time. We instantiate our guarantees for policies parameterized by diffusion models and prove that if the learner accurately estimates the score of the (noise-augmented) expert policy, then the distribution of imitator trajectories is close to the demonstrator distribution in a natural optimal transport distance. Our analysis constructs intricate couplings between noise-augmented trajectories, a technique that may be of independent interest. We conclude by empirically validating our algorithmic recommendations, and discussing implications for future research directions for better behavior cloning with generative modeling.