DynaGuide: Steering Diffusion Polices with Active Dynamic Guidance

TL;DR

DynaGuide introduces a novel method for steering diffusion policies using an external dynamics model, enabling flexible, robust, and multi-objective control of complex policies in real-world scenarios.

Contribution

It presents DynaGuide, a new approach that separates the dynamics guidance from the base policy, allowing for versatile steering of pretrained diffusion policies without retraining.

Findings

Achieved 70% success rate in articulated CALVIN tasks.

Outperformed goal-conditioning by 5.4x with low-quality objectives.

Successfully steered real robot policies to express preferences and create new behaviors.

Abstract

Deploying large, complex policies in the real world requires the ability to steer them to fit the needs of a situation. Most common steering approaches, like goal-conditioning, require training the robot policy with a distribution of test-time objectives in mind. To overcome this limitation, we present DynaGuide, a steering method for diffusion policies using guidance from an external dynamics model during the diffusion denoising process. DynaGuide separates the dynamics model from the base policy, which gives it multiple advantages, including the ability to steer towards multiple objectives, enhance underrepresented base policy behaviors, and maintain robustness on low-quality objectives. The separate guidance signal also allows DynaGuide to work with off-the-shelf pretrained diffusion policies. We demonstrate the performance and features of DynaGuide against other steering approaches in a series of simulated and real experiments, showing an average steering success of 70% on a set of articulated CALVIN tasks and outperforming goal-conditioning by 5.4x when steered with low-quality objectives. We also successfully steer an off-the-shelf real robot policy to express preference for particular objects and even create novel behavior. Videos and more can be found on the project website: https://dynaguide.github.io