DreamControl-v2: Simpler and Scalable Autonomous Humanoid Skills via Trainable Guided Diffusion Priors

TL;DR

This paper introduces an improved, scalable framework for autonomous humanoid skills using guided diffusion models trained directly in robot motion space, enhancing skill diversity and automation.

Contribution

It proposes training a guided diffusion model directly in the robot's motion space, integrating diverse datasets, and scaling trajectory generation for better RL policy robustness.

Findings

Larger training data mixture captures a wider range of skills.

Automated pipeline removes manual filtering.

Scaling trajectory generation improves RL policy robustness.

Abstract

Developing robust autonomous loco-manipulation skills for humanoids remains an open problem in robotics. While RL has been applied successfully to legged locomotion, applying it to complex, interaction-rich manipulation tasks is harder given long-horizon planning challenges for manipulation. A recent approach along these lines is DreamControl, which addresses these issues by leveraging off-the-shelf human motion diffusion models as a generative prior to guide RL policies during training. In this paper, we investigate the impact of DreamControl's motion prior and propose an improved framework that trains a guided diffusion model directly in the humanoid robot's motion space, aggregating diverse human and robot datasets into a unified embodiment space. We demonstrate that our approach captures a wider range of skills due to the larger training data mixture and establishes a more automated pipeline by removing the need for manual filtering interventions. Furthermore, we show that scaling the generation of reference trajectories is important for achieving robust downstream RL policies. We validate our approach through extensive experiments in simulation and on a real Unitree-G1.