Scaling Cross-Environment Failure Reasoning Data for Vision-Language Robotic Manipulation

TL;DR

This paper introduces FailCoT, a large-scale dataset for robotic failure reasoning, and Guardian, a vision-language model that improves failure detection and task success in robotic manipulation by leveraging this data.

Contribution

The paper presents a novel automatic framework for generating diverse robotic failure data and a new model, Guardian, that achieves state-of-the-art results in failure reasoning and task success.

Findings

Guardian outperforms previous models on three real-world benchmarks.

Scaling failure reasoning data significantly improves generalization in robotic failure detection.

Integration with LLM-based policies enhances task success rates in real-world robot deployment.

Abstract

Robust robotic manipulation requires reliable failure detection and recovery. Although recent Vision-Language Models (VLMs) show promise in robot failure detection, their generalization is severely limited by the scarcity and narrow coverage of failure data. To address this bottleneck, we propose an automatic framework for generating diverse robotic planning and execution failures across both simulated and real-world environments. Our approach perturbs successful manipulation trajectories to synthesize failures that reflect realistic failure distributions, and leverages VLMs to produce structured step-by-step reasoning traces. This yields FailCoT, a large-scale failure reasoning dataset built upon the RLBench simulator and the BridgeDataV2 real-robot dataset. Using FailCoT, we train Guardian, a multi-view reasoning VLM for unified planning and execution verification. Guardian achieves state-of-the-art performance on three unseen real-world benchmarks: RoboFail, RoboVQA, and our newly introduced UR5-Fail. When integrated with a state-of-the-art LLM-based manipulation policy, it consistently boosts task success rates in both simulation and real-world deployment. These results demonstrate that scaling high-quality failure reasoning data is critical for improving generalization in robotic failure detection. Code, Data, and Models available at https://www.di.ens.fr/willow/research/guardian/.