Learning to Reach Goals via Diffusion

TL;DR

This paper introduces Merlin, a diffusion-based goal-conditioned reinforcement learning method that efficiently reaches goals without value functions, demonstrating superior performance and computational efficiency in offline tasks.

Contribution

Merlin is the first diffusion in state space approach for RL that requires only one denoising step per environment step, improving efficiency and effectiveness.

Findings

Outperforms state-of-the-art offline goal-reaching methods

Requires only one denoising iteration per environment step

Significantly improves computational efficiency over previous diffusion-based RL methods

Abstract

We present a novel perspective on goal-conditioned reinforcement learning by framing it within the context of denoising diffusion models. Analogous to the diffusion process, where Gaussian noise is used to create random trajectories that walk away from the data manifold, we construct trajectories that move away from potential goal states. We then learn a goal-conditioned policy to reverse these deviations, analogous to the score function. This approach, which we call Merlin, can reach specified goals from arbitrary initial states without learning a separate value function. In contrast to recent works utilizing diffusion models in offline RL, Merlin stands out as the first method to perform diffusion in the state space, requiring only one ``denoising" iteration per environment step. We experimentally validate our approach in various offline goal-reaching tasks, demonstrating substantial performance enhancements compared to state-of-the-art methods while improving computational efficiency over other diffusion-based RL methods by an order of magnitude. Our results suggest that this perspective on diffusion for RL is a simple and scalable approach for sequential decision making.