VADF: Vision-Adaptive Diffusion Policy Framework for Efficient Robotic Manipulation

TL;DR

VADF introduces a vision-driven adaptive framework for diffusion policies in robotic manipulation, enhancing training efficiency and inference success by prioritizing difficult samples and adaptively segmenting tasks based on visual input.

Contribution

The paper presents a novel dual-adaptive framework with ALN for training and HVTS for inference, improving convergence speed and early success in robotic diffusion policies.

Findings

Reduces training convergence steps significantly.

Improves early inference success rate.

Decreases computational overhead during inference.

Abstract

Diffusion policies are becoming mainstream in robotic manipulation but suffer from hard negative class imbalance due to uniform sampling and lack of sample difficulty awareness, leading to slow training convergence and frequent inference timeout failures. We propose VADF (Vision-Adaptive Diffusion Policy Framework), a vision-driven dual-adaptive framework that significantly reduces convergence steps and achieves early success in inference, with model-agnostic design enabling seamless integration into any diffusion policy architecture. During training, we introduce Adaptive Loss Network (ALN), a lightweight MLP-based loss predictor that quantifies per-step sample difficulty in real time. Guided by hard negative mining, it performs weighted sampling to prioritize high-loss regions, enabling adaptive weight updates and faster convergence. In inference, we design the Hierarchical Vision Task Segmenter (HVTS), which decomposes high-level task instructions into multi-stage low-level sub-instructions based on visual input. It adaptively segments action sequences into simple and complex subtasks by assigning shorter noise schedules with longer direct execution sequences to simple actions, and longer noise steps with shorter execution sequences to complex ones, thereby dramatically reducing computational overhead and significantly improving the early success rate.