Referring-Aware Visuomotor Policy Learning for Closed-Loop Manipulation

TL;DR

This paper introduces ReV, a closed-loop visuomotor policy framework that adaptively re-plans trajectories in real-time for robotic manipulation, enhancing robustness to unforeseen circumstances without extra data.

Contribution

ReV is a novel framework that integrates sparse referring points into visuomotor policies for dynamic, real-time trajectory adaptation during manipulation tasks.

Findings

ReV achieves higher success rates in simulated tasks.

ReV demonstrates robustness in real-world manipulation scenarios.

ReV operates without additional data or fine-tuning.

Abstract

This paper addresses a fundamental problem of visuomotor policy learning for robotic manipulation: how to enhance robustness in out-of-distribution execution errors or dynamically re-routing trajectories, where the model relies solely on the original expert demonstrations for training. We introduce the Referring-Aware Visuomotor Policy (ReV), a closed-loop framework that can adapt to unforeseen circumstances by instantly incorporating sparse referring points provided by a human or a high-level reasoning planner. Specifically, ReV leverages the coupled diffusion heads to preserve standard task execution patterns while seamlessly integrating sparse referring via a trajectory-steering strategy. Upon receiving a specific referring point, the global diffusion head firstly generates a sequence of globally consistent yet temporally sparse action anchors, while identifies the precise temporal position for the referring point within this sequence. Subsequently, the local diffusion head adaptively interpolates adjacent anchors based on the current temporal position for specific tasks. This closed-loop process repeats at every execution step, enabling real-time trajectory replanning in response to dynamic changes in the scene. In practice, rather than relying on elaborate annotations, ReV is trained only by applying targeted perturbations to expert demonstrations. Without any additional data or fine-tuning scheme, ReV achieve higher success rates across challenging simulated and real-world tasks.