RoboFAC: A Comprehensive Framework for Robotic Failure Analysis and Correction

TL;DR

RoboFAC introduces a large-scale failure-centric dataset and a specialized multimodal model to improve failure diagnosis and recovery in robotic manipulation, significantly enhancing robustness and efficiency in real-world scenarios.

Contribution

The paper presents RoboFAC, a novel framework with a large failure dataset and a lightweight model for failure analysis and correction in robotic manipulation tasks.

Findings

RoboFAC achieves 34.1% higher failure analysis accuracy than GPT-4o.

Integration of RoboFAC improves task success by 29.1% in real-world experiments.

The framework reduces latency compared to large proprietary models.

Abstract

Vision-Language-Action (VLA) models have recently advanced robotic manipulation by translating natural-language instructions and visual observations into control actions. However, existing VLAs are primarily trained on successful expert demonstrations and lack structured supervision for failure diagnosis and recovery, limiting robustness in open-world scenarios. To address this limitation, we propose the Robotic Failure Analysis and Correction (RoboFAC) framework. We construct a large-scale failure-centric dataset comprising 9,440 erroneous manipulation trajectories and 78,623 QA pairs across 53 scenes in both simulation and real-world environments, with systematically categorized failure types. Leveraging this dataset, we develop a lightweight multimodal model specialized for task understanding, failure analysis, and failure correction, enabling efficient local deployment while remaining competitive with large proprietary models. Experimental results demonstrate that RoboFAC achieves a 34.1% higher failure analysis accuracy compared to GPT-4o. Furthermore, we integrated RoboFAC as an external supervisor in a real-world VLA control pipeline, yielding a 29.1% relative improvement across four tasks while significantly reducing latency relative to GPT-4o. These results demonstrate that RoboFAC enables systematic failure diagnosis and recovery, significantly enhancing VLA recovery capabilities. Our model and dataset are publicly available at https://github.com/MINT-SJTU/RoboFAC.