Predictive Red Teaming: Breaking Policies Without Breaking Robots

TL;DR

This paper introduces RoboART, an automated pipeline for predicting visuomotor policy vulnerabilities to environmental changes, enabling efficient identification of failure scenarios without hardware testing.

Contribution

The paper presents RoboART, a novel generative image editing and anomaly detection framework for predictive red teaming of visuomotor policies in off-nominal conditions.

Findings

RoboART predicts performance degradation with less than 0.19 average difference from real success rates.

Targeted data collection based on RoboART predictions significantly improves policy performance.

The approach reduces the need for costly hardware evaluations in vulnerability assessment.

Abstract

Visuomotor policies trained via imitation learning are capable of performing challenging manipulation tasks, but are often extremely brittle to lighting, visual distractors, and object locations. These vulnerabilities can depend unpredictably on the specifics of training, and are challenging to expose without time-consuming and expensive hardware evaluations. We propose the problem of predictive red teaming: discovering vulnerabilities of a policy with respect to environmental factors, and predicting the corresponding performance degradation without hardware evaluations in off-nominal scenarios. In order to achieve this, we develop RoboART: an automated red teaming (ART) pipeline that (1) modifies nominal observations using generative image editing to vary different environmental factors, and (2) predicts performance under each variation using a policy-specific anomaly detector executed on edited observations. Experiments across 500+ hardware trials in twelve off-nominal conditions for visuomotor diffusion policies demonstrate that RoboART predicts performance degradation with high accuracy (less than 0.19 average difference between predicted and real success rates). We also demonstrate how predictive red teaming enables targeted data collection: fine-tuning with data collected under conditions predicted to be adverse boosts baseline performance by 2-7x.