Self-Refining Vision Language Model for Robotic Failure Detection and Reasoning

TL;DR

This paper introduces ARMOR, a multi-task self-refinement model that improves robotic failure detection and reasoning by leveraging heterogeneous supervision and iterative prediction, achieving state-of-the-art results in diverse environments.

Contribution

The paper presents ARMOR, a novel adaptive multi-task model that enhances robotic failure detection and reasoning through iterative self-refinement and heterogeneous supervision learning.

Findings

Up to 30% improvement in failure detection rate

Up to 100% improvement in reasoning accuracy

Robust performance across diverse environments

Abstract

Reasoning about failures is crucial for building reliable and trustworthy robotic systems. Prior approaches either treat failure reasoning as a closed-set classification problem or assume access to ample human annotations. Failures in the real world are typically subtle, combinatorial, and difficult to enumerate, whereas rich reasoning labels are expensive to acquire. We address this problem by introducing ARMOR: Adaptive Round-based Multi-task mOdel for Robotic failure detection and reasoning. We formulate detection and reasoning as a multi-task self-refinement process, where the model iteratively predicts detection outcomes and natural language reasoning conditioned on past outputs. During training, ARMOR learns from heterogeneous supervision - large-scale sparse binary labels and small-scale rich reasoning annotations - optimized via a combination of offline and online imitation learning. At inference time, ARMOR generates multiple refinement trajectories and selects the most confident prediction via a self-certainty metric. Experiments across diverse environments show that ARMOR achieves state-of-the-art performance by improving over the previous approaches by up to 30% on failure detection rate and up to 100% in reasoning measured through LLM fuzzy match score, demonstrating robustness to heterogeneous supervision and open-ended reasoning beyond predefined failure modes. We provide dditional visualizations on our website: https://sites.google.com/utexas.edu/armor