Diff-DAgger: Uncertainty Estimation with Diffusion Policy for Robotic Manipulation

TL;DR

Diff-DAgger introduces an uncertainty estimation method for diffusion policies in robotic manipulation, enhancing out-of-distribution failure detection and improving task success rates through active expert querying.

Contribution

It presents Diff-DAgger, a novel robot-gated DAgger algorithm that effectively estimates uncertainty in expressive diffusion policies, addressing limitations of existing methods like Ensemble-DAgger.

Findings

Improves task failure prediction accuracy by 39%.

Increases task completion rate by 20.6%.

Reduces training time by a factor of 7.8.

Abstract

Recently, diffusion policy has shown impressive results in handling multi-modal tasks in robotic manipulation. However, it has fundamental limitations in out-of-distribution failures that persist due to compounding errors and its limited capability to extrapolate. One way to address these limitations is robot-gated DAgger, an interactive imitation learning with a robot query system to actively seek expert help during policy rollout. While robot-gated DAgger has high potential for learning at scale, existing methods like Ensemble-DAgger struggle with highly expressive policies: They often misinterpret policy disagreements as uncertainty at multi-modal decision points. To address this problem, we introduce Diff-DAgger, an efficient robot-gated DAgger algorithm that leverages the training objective of diffusion policy. We evaluate Diff-DAgger across different robot tasks including stacking, pushing, and plugging, and show that Diff-DAgger improves the task failure prediction by 39.0%, the task completion rate by 20.6%, and reduces the wall-clock time by a factor of 7.8. We hope that this work opens up a path for efficiently incorporating expressive yet data-hungry policies into interactive robot learning settings. The project website is available at: https://diffdagger.github.io.