Diffusion Model Predictive Control

TL;DR

This paper introduces Diffusion Model Predictive Control (D-MPC), a new MPC method that leverages diffusion models for multi-step action proposals and dynamics modeling, demonstrating superior performance and adaptability in benchmark tests.

Contribution

The paper presents a novel MPC approach using diffusion models for multi-step action and dynamics prediction, improving offline planning and real-time adaptation.

Findings

Outperforms existing model-based offline planning methods on D4RL benchmark

Achieves competitive results with state-of-the-art reinforcement learning methods

Demonstrates ability to optimize new reward functions and adapt to new dynamics

Abstract

We propose Diffusion Model Predictive Control (D-MPC), a novel MPC approach that learns a multi-step action proposal and a multi-step dynamics model, both using diffusion models, and combines them for use in online MPC. On the popular D4RL benchmark, we show performance that is significantly better than existing model-based offline planning methods using MPC (e.g. MBOP) and competitive with state-of-the-art (SOTA) model-based and model-free reinforcement learning methods. We additionally illustrate D-MPC's ability to optimize novel reward functions at run time and adapt to novel dynamics, and highlight its advantages compared to existing diffusion-based planning baselines.