Automated Testing with Temporal Logic Specifications for Robotic Controllers using Adaptive Experiment Design

TL;DR

This paper introduces an adaptive experiment design method that uses temporal logic specifications and surrogate modeling to efficiently evaluate robotic control system robustness, especially when dealing with black-box components and large environment spaces.

Contribution

It combines smooth robustness metrics with adaptive experiment design to select the most informative tests, enabling efficient robustness assessment without exhaustive testing.

Findings

Sample-efficient robustness descriptions achieved

Effective assessment under domain shift demonstrated

Surrogate model guides experiment selection

Abstract

Many robot control scenarios involve assessing system robustness against a task specification. If either the controller or environment are composed of "black-box" components with unknown dynamics, we cannot rely on formal verification to assess our system. Assessing robustness via exhaustive testing is also often infeasible if the space of environments is large compared to experiment cost. Given limited budget, we provide a method to choose experiment inputs which give greatest insight into system performance against a given specification across the domain. By combining smooth robustness metrics for signal temporal logic with techniques from adaptive experiment design, our method chooses the most informative experimental inputs by incrementally constructing a surrogate model of the specification robustness. This model then chooses the next experiment to be in an area where there is either high prediction error or uncertainty. Our experiments show how this adaptive experimental design technique results in sample-efficient descriptions of system robustness. Further, we show how to use the model built via the experiment design process to assess the behaviour of a data-driven control system under domain shift.