Closed-Loop Visuomotor Control with Generative Expectation for Robotic Manipulation

TL;DR

This paper introduces CLOVER, a closed-loop visuomotor control framework for robots that uses visual feedback and generative models to improve long-horizon task performance, achieving state-of-the-art results.

Contribution

The paper presents a novel closed-loop control system combining visual generative models, measurable embedding spaces, and feedback-driven planning for robotic manipulation.

Findings

Achieves 8% improvement over open-loop methods on CALVIN benchmark.

Demonstrates effective real-world robotic task performance.

Introduces a new framework integrating visual diffusion models with feedback control.

Abstract

Despite significant progress in robotics and embodied AI in recent years, deploying robots for long-horizon tasks remains a great challenge. Majority of prior arts adhere to an open-loop philosophy and lack real-time feedback, leading to error accumulation and undesirable robustness. A handful of approaches have endeavored to establish feedback mechanisms leveraging pixel-level differences or pre-trained visual representations, yet their efficacy and adaptability have been found to be constrained. Inspired by classic closed-loop control systems, we propose CLOVER, a closed-loop visuomotor control framework that incorporates feedback mechanisms to improve adaptive robotic control. CLOVER consists of a text-conditioned video diffusion model for generating visual plans as reference inputs, a measurable embedding space for accurate error quantification, and a feedback-driven controller that refines actions from feedback and initiates replans as needed. Our framework exhibits notable advancement in real-world robotic tasks and achieves state-of-the-art on CALVIN benchmark, improving by 8% over previous open-loop counterparts. Code and checkpoints are maintained at https://github.com/OpenDriveLab/CLOVER.