Addressing Failures in Robotics using Vision-Based Language Models (VLMs) and Behavior Trees (BT)

TL;DR

This paper presents a novel approach combining Vision Language Models and Behavior Trees to detect, identify, and recover from both known and unknown failures in robotic systems, enhancing autonomy and robustness.

Contribution

It introduces the integration of VLMs with BTs for failure detection and recovery, enabling autonomous handling of unforeseen failures in robotics.

Findings

Effective failure detection using VLMs in simulations

Successful incorporation of VLM-generated conditions into BTs

Improved robustness in robotic failure management

Abstract

In this paper, we propose an approach that combines Vision Language Models (VLMs) and Behavior Trees (BTs) to address failures in robotics. Current robotic systems can handle known failures with pre-existing recovery strategies, but they are often ill-equipped to manage unknown failures or anomalies. We introduce VLMs as a monitoring tool to detect and identify failures during task execution. Additionally, VLMs generate missing conditions or skill templates that are then incorporated into the BT, ensuring the system can autonomously address similar failures in future tasks. We validate our approach through simulations in several failure scenarios.