ROVER: Regulator-Driven Robust Temporal Verification of Black-Box Robot Policies

TL;DR

This paper introduces ROVER, a regulator-driven framework for verifying black-box robot policies against temporal safety specifications, improving safety adherence through iterative retraining and validation in virtual and real-world scenarios.

Contribution

We propose a regulator-in-the-loop approach that uses temporal logic metrics to guide verification and retraining of black-box policies, enhancing safety compliance.

Findings

Regulator-guided retraining increased satisfaction rates by 43.8%.

Improved average performance (TRV) and reduced violations (LRV).

27% improvement in smooth navigation on real robots.

Abstract

We present a novel, regulator-driven approach for the temporal verification of black-box autonomous robot policies, inspired by real-world certification processes where regulators often evaluate observable behavior without access to model internals. Central to our method is a regulator-in-the-loop approach that evaluates execution traces from black-box policies against temporal safety requirements. These requirements, expressed as prioritized Signal Temporal Logic (STL) specifications, characterize behavior changes over time and encode domain knowledge into the verification process. We use Total Robustness Value (TRV) and Largest Robustness Value (LRV) to quantify average performance and worst-case adherence, and introduce Average Violation Robustness Value (AVRV) to measure average specification violation. Together, these metrics guide targeted retraining and iterative model improvement. Our approach accommodates diverse temporal safety requirements (e.g., lane-keeping, delayed acceleration, and turn smoothness), capturing persistence, sequencing, and response across two distinct domains (virtual racing game and mobile robot navigation). Across six STL specifications in both scenarios, regulator-guided retraining increased satisfaction rates by an average of 43.8%, with consistent improvement in average performance (TRV) and reduced violation severity (LRV) in half of the specifications. Finally, real-world validation on a TurtleBot3 robot demonstrates a 27% improvement in smooth-navigation satisfaction, yielding smoother paths and stronger compliance with STL-defined temporal safety requirements.