Failure Prediction from Limited Hardware Demonstrations

TL;DR

This paper introduces a three-step methodology for predicting failures in robotic systems using limited real-world demonstrations combined with model-based testing, reducing the need for extensive and risky data collection.

Contribution

It proposes a novel approach that integrates model simulations and Bayesian inference to efficiently discover failures with minimal true system demonstrations.

Findings

Effective failure prediction with limited demonstrations

Successful application to robotic arm pushing task

Demonstrated failure discovery in racing car scenario

Abstract

Prediction of failures in real-world robotic systems either requires accurate model information or extensive testing. Partial knowledge of the system model makes simulation-based failure prediction unreliable. Moreover, obtaining such demonstrations is expensive, and could potentially be risky for the robotic system to repeatedly fail during data collection. This work presents a novel three-step methodology for discovering failures that occur in the true system by using a combination of a limited number of demonstrations from the true system and the failure information processed through sampling-based testing of a model dynamical system. Given a limited budget $N$ of demonstrations from true system and a model dynamics (with potentially large modeling errors), the proposed methodology comprises of a) exhaustive simulations for discovering algorithmic failures using the model dynamics; b) design of initial $N_1$ demonstrations of the true system using Bayesian inference to learn a Gaussian process regression (GPR)-based failure predictor; and c) iterative $N - N_1$ demonstrations of the true system for updating the failure predictor. To illustrate the efficacy of the proposed methodology, we consider: a) the failure discovery for the task of pushing a T block to a fixed target region with UR3E collaborative robot arm using a diffusion policy; and b) the failure discovery for an F1-Tenth racing car tracking a given raceline under an LQR control policy.