Parallel and Multi-Objective Falsification with Scenic and VerifAI

TL;DR

This paper enhances simulation-based falsification of autonomous systems by introducing parallelism and multi-objective optimization in Scenic and VerifAI, significantly improving scalability and effectiveness.

Contribution

It presents a parallelized framework and multi-objective extension for falsification, enabling faster and more comprehensive testing of autonomous systems.

Findings

Reduced execution time through parallelism

Effective multi-objective falsification using rulebooks

Improved scalability demonstrated on Scenic benchmarks

Abstract

Falsification has emerged as an important tool for simulation-based verification of autonomous systems. In this paper, we present extensions to the Scenic scenario specification language and VerifAI toolkit that improve the scalability of sampling-based falsification methods by using parallelism and extend falsification to multi-objective specifications. We first present a parallelized framework that is interfaced with both the simulation and sampling capabilities of Scenic and the falsification capabilities of VerifAI, reducing the execution time bottleneck inherently present in simulation-based testing. We then present an extension of VerifAI's falsification algorithms to support multi-objective optimization during sampling, using the concept of rulebooks to specify a preference ordering over multiple metrics that can be used to guide the counterexample search process. Lastly, we evaluate the benefits of these extensions with a comprehensive set of benchmarks written in the Scenic language.