Diffusion Models as Optimizers for Efficient Planning in Offline RL

TL;DR

This paper introduces the Trajectory Diffuser, a method that decomposes diffusion model sampling into trajectory generation and optimization, significantly improving planning efficiency in offline reinforcement learning without sacrificing performance.

Contribution

It proposes a novel decomposition approach and a hybrid model combining autoregressive and diffusion techniques for faster, high-quality offline RL planning.

Findings

Achieves 3-10x faster inference speed than previous methods.

Outperforms existing sequence modeling approaches in overall performance.

Demonstrates effectiveness on D4RL benchmarks.

Abstract

Diffusion models have shown strong competitiveness in offline reinforcement learning tasks by formulating decision-making as sequential generation. However, the practicality of these methods is limited due to the lengthy inference processes they require. In this paper, we address this problem by decomposing the sampling process of diffusion models into two decoupled subprocesses: 1) generating a feasible trajectory, which is a time-consuming process, and 2) optimizing the trajectory. With this decomposition approach, we are able to partially separate efficiency and quality factors, enabling us to simultaneously gain efficiency advantages and ensure quality assurance. We propose the Trajectory Diffuser, which utilizes a faster autoregressive model to handle the generation of feasible trajectories while retaining the trajectory optimization process of diffusion models. This allows us to achieve more efficient planning without sacrificing capability. To evaluate the effectiveness and efficiency of the Trajectory Diffuser, we conduct experiments on the D4RL benchmarks. The results demonstrate that our method achieves $\it 3$-$\it 10 \times$ faster inference speed compared to previous sequence modeling methods, while also outperforming them in terms of overall performance. https://github.com/RenMing-Huang/TrajectoryDiffuser Keywords: Reinforcement Learning and Efficient Planning and Diffusion Model