A Review of Online Diffusion Policy RL Algorithms for Scalable Robotic Control

TL;DR

This paper reviews and empirically analyzes online diffusion policy reinforcement learning algorithms for robotic control, categorizing approaches, evaluating their performance across diverse tasks, and identifying key trade-offs and bottlenecks.

Contribution

It introduces a novel taxonomy of online DPRL algorithms, provides a comprehensive benchmark on 12 robotic tasks, and offers practical guidelines and future directions for scalable robotic learning.

Findings

Trade-offs between sample efficiency and scalability in different algorithm families

Identification of computational bottlenecks limiting real-world deployment

Insights into generalization and robustness across diverse robotic tasks

Abstract

Diffusion policies have emerged as a powerful approach for robotic control, demonstrating superior expressiveness in modeling multimodal action distributions compared to conventional policy networks. However, their integration with online reinforcement learning remains challenging due to fundamental incompatibilities between diffusion model training objectives and standard RL policy improvement mechanisms. This paper presents the first comprehensive review and empirical analysis of current Online Diffusion Policy Reinforcement Learning (Online DPRL) algorithms for scalable robotic control systems. We propose a novel taxonomy that categorizes existing approaches into four distinct families--Action-Gradient, Q-Weighting, Proximity-Based, and Backpropagation Through Time (BPTT) methods--based on their policy improvement mechanisms. Through extensive experiments on a unified NVIDIA Isaac Lab benchmark encompassing 12 diverse robotic tasks, we systematically evaluate representative algorithms across five critical dimensions: task diversity, parallelization capability, diffusion step scalability, cross-embodiment generalization, and environmental robustness. Our analysis identifies key findings regarding the fundamental trade-offs inherent in each algorithmic family, particularly concerning sample efficiency and scalability. Furthermore, we reveal critical computational and algorithmic bottlenecks that currently limit the practical deployment of online DPRL. Based on these findings, we provide concrete guidelines for algorithm selection tailored to specific operational constraints and outline promising future research directions to advance the field toward more general and scalable robotic learning systems.