RACER: Rich Language-Guided Failure Recovery Policies for Imitation Learning

TL;DR

RACER introduces a scalable framework combining rich language annotations and failure recovery data to improve robotic visuomotor policies, enabling better error correction and task success in diverse environments.

Contribution

The paper presents RACER, a novel supervisor-actor framework that integrates a vision-language model with failure recovery data to enhance robot control and robustness.

Findings

RACER outperforms state-of-the-art methods on RLbench tasks.

RACER demonstrates superior zero-shot generalization to unseen tasks.

RACER achieves effective real-world robot manipulation performance.

Abstract

Developing robust and correctable visuomotor policies for robotic manipulation is challenging due to the lack of self-recovery mechanisms from failures and the limitations of simple language instructions in guiding robot actions. To address these issues, we propose a scalable data generation pipeline that automatically augments expert demonstrations with failure recovery trajectories and fine-grained language annotations for training. We then introduce Rich languAge-guided failure reCovERy (RACER), a supervisor-actor framework, which combines failure recovery data with rich language descriptions to enhance robot control. RACER features a vision-language model (VLM) that acts as an online supervisor, providing detailed language guidance for error correction and task execution, and a language-conditioned visuomotor policy as an actor to predict the next actions. Our experimental results show that RACER outperforms the state-of-the-art Robotic View Transformer (RVT) on RLbench across various evaluation settings, including standard long-horizon tasks, dynamic goal-change tasks and zero-shot unseen tasks, achieving superior performance in both simulated and real world environments. Videos and code are available at: https://rich-language-failure-recovery.github.io.