Diagnose, Correct, and Learn from Manipulation Failures via Visual Symbols

TL;DR

This paper introduces ViFailback, a comprehensive framework and dataset for diagnosing and correcting robotic manipulation failures using visual symbols, improving real-world failure recovery in vision-language models.

Contribution

The paper presents ViFailback, a novel framework with a large-scale dataset and benchmark for failure diagnosis and correction in robotic manipulation, enhancing real-world applicability and model performance.

Findings

ViFailback-Bench enables detailed evaluation of VLMs in failure scenarios.

ViFailback-8B significantly improves failure diagnosis and correction performance.

Integration with VLA models helps robots recover from manipulation failures.

Abstract

Vision-Language-Action (VLA) models have recently achieved remarkable progress in robotic manipulation, yet they remain limited in failure diagnosis and learning from failures. Additionally, existing failure datasets are mostly generated programmatically in simulation, which limits their generalization to the real world. In light of these, we introduce ViFailback, a framework designed to diagnose robotic manipulation failures and provide both textual and visual correction guidance. Our framework utilizes explicit visual symbols to enhance annotation efficiency. We further release the ViFailback dataset, a large-scale collection of 58,126 Visual Question Answering (VQA) pairs along with their corresponding 5,202 real-world manipulation trajectories. Based on the dataset, we establish ViFailback-Bench, a benchmark of 11 fine-grained VQA tasks designed to assess the failure diagnosis and correction abilities of Vision-Language Models (VLMs), featuring ViFailback-Bench Lite for closed-ended and ViFailback-Bench Hard for open-ended evaluation. To demonstrate the effectiveness of our framework, we built the ViFailback-8B VLM, which not only achieves significant overall performance improvement on ViFailback-Bench but also generates visual symbols for corrective action guidance. Finally, by integrating ViFailback-8B with a VLA model, we conduct real-world robotic experiments demonstrating its ability to assist the VLA model in recovering from failures. Project Website: https://x1nyuzhou.github.io/vifailback.github.io/