Vision Language Model-based Testing of Industrial Autonomous Mobile Robots

TL;DR

This paper introduces a Vision Language Model-based testing method for industrial autonomous mobile robots, enabling the generation of diverse, requirement-violating human interaction scenarios in simulation to improve safety and robustness.

Contribution

It presents a novel VLM-based testing approach (RVSG) that effectively generates diverse, requirement-violating scenarios for industrial AMRs in simulation environments.

Findings

RVSG effectively generates requirement-violating scenarios.

Scenarios increase variability in robot behavior.

Method helps reveal uncertain behaviors of AMRs.

Abstract

PAL Robotics, in Spain, builds a variety of Autonomous Mobile Robots (AMRs), which are deployed in diverse environments (e.g., warehouses, retail spaces, and offices), where they work alongside humans. Given that human behavior can be unpredictable and that AMRs may not have been trained to handle all possible unknown and uncertain behaviors, it is important to test AMRs under a wide range of human interactions to ensure their safe behavior. Moreover, testing in real environments with actual AMRs and humans is often costly, impractical, and potentially hazardous (e.g., it could result in human injury). To this end, we propose a Vision Language Model (VLM)-based testing approach (RVSG) for industrial AMRs developed together with PAL Robotics. Based on the functional and safety requirements, RVSG uses the VLM to generate diverse human behaviors that violate these requirements. We evaluated RVSG with several requirements and navigation routes in a simulator using the latest AMR from PAL Robotics. Our results show that, compared with the baseline, RVSG can effectively generate requirement-violating scenarios. Moreover, RVSG-generated scenarios increase variability in robot behavior, thereby helping reveal their uncertain behaviors.