Boosting Continuous Control with Consistency Policy

TL;DR

This paper introduces CPQL, a novel method that enhances diffusion model-based offline and online reinforcement learning by enabling single-step action derivation, significantly improving efficiency and guidance accuracy.

Contribution

The paper proposes CPQL, a time-efficient policy method that maps reverse diffusion trajectories to actions, addressing speed and guidance issues in diffusion-based reinforcement learning.

Findings

Achieves state-of-the-art performance on 11 offline tasks.

Improves inference speed by nearly 45 times.

Seamlessly extends to online RL tasks.

Abstract

Due to its training stability and strong expression, the diffusion model has attracted considerable attention in offline reinforcement learning. However, several challenges have also come with it: 1) The demand for a large number of diffusion steps makes the diffusion-model-based methods time inefficient and limits their applications in real-time control; 2) How to achieve policy improvement with accurate guidance for diffusion model-based policy is still an open problem. Inspired by the consistency model, we propose a novel time-efficiency method named Consistency Policy with Q-Learning (CPQL), which derives action from noise by a single step. By establishing a mapping from the reverse diffusion trajectories to the desired policy, we simultaneously address the issues of time efficiency and inaccurate guidance when updating diffusion model-based policy with the learned Q-function. We demonstrate that CPQL can achieve policy improvement with accurate guidance for offline reinforcement learning, and can be seamlessly extended for online RL tasks. Experimental results indicate that CPQL achieves new state-of-the-art performance on 11 offline and 21 online tasks, significantly improving inference speed by nearly 45 times compared to Diffusion-QL. We will release our code later.