ImitDiff: Transferring Foundation-Model Priors for Distraction Robust Visuomotor Policy

TL;DR

ImitDiff is a diffusion-based visuomotor policy that uses semantic masks from foundation models to improve robot manipulation robustness in complex and distracting visual environments.

Contribution

We introduce ImitDiff, a novel dual-resolution, diffusion-guided imitation learning framework leveraging vision-language priors for distraction-robust robot manipulation.

Findings

Outperforms state-of-the-art methods in complex scenes.

Shows strong zero-shot generalization to new objects and distractions.

Achieves faster inference with a new diffusion transformer action head.

Abstract

Visuomotor imitation learning policies enable robots to efficiently acquire manipulation skills from visual demonstrations. However, as scene complexity and visual distractions increase, policies that perform well in simple settings often experience substantial performance degradation. To address this challenge, we propose ImitDiff, a diffusion-based imitation learning policy guided by fine-grained semantics within a dual-resolution workflow. Leveraging pretrained priors of vision-language foundation models, our method transforms high-level instructions into pixel-level visual semantic masks. These masks guide a dual-resolution perception pipeline that captures both global context (e.g., overall layout) from low-resolution observation and fine-grained local features (e.g., geometric details) from high-resolution observation, enabling the policy to focus on task-relevant regions. Additionally, we introduce a consistency-driven diffusion transformer action head that bridges visual semantic conditions and real-time action generation. Extensive experiments demonstrate that ImitDiff outperforms state-of-the-art vision-language manipulation frameworks, as well as visuomotor imitation learning policies, particularly under increased scene complexity and visual distractions. Notably, ImitDiff exhibits strong generalization in zero-shot settings involving novel objects and visual distractions. Furthermore, our consistency-driven action head achieves an order-of-magnitude improvement in inference speed while maintaining competitive success rates.