Paracosm: A Language and Tool for Testing Autonomous Driving Systems

TL;DR

Paracosm is a reactive language and tool that enables systematic, realistic testing of autonomous driving systems in complex scenarios, using simulation, combinatorial testing, and automatic test case generation.

Contribution

It introduces a novel language and infrastructure for modeling and testing autonomous vehicles in realistic, complex environments with automated test generation.

Findings

Paracosm can expose incorrect behaviors in autonomous driving systems.

The tool effectively explores the state space of driving scenarios.

It demonstrates modeling and testing capabilities on neural network-based systems.

Abstract

Systematic testing of autonomous vehicles operating in complex real-world scenarios is a difficult and expensive problem. We present Paracosm, a reactive language for writing test scenarios for autonomous driving systems. Paracosm allows users to programmatically describe complex driving situations with specific visual features, e.g., road layout in an urban environment, as well as reactive temporal behaviors of cars and pedestrians. Paracosm programs are executed on top of a game engine that provides realistic physics simulation and visual rendering. The infrastructure allows systematic exploration of the state space, both for visual features (lighting, shadows, fog) and for reactive interactions with the environment (pedestrians, other traffic). We define a notion of test coverage for Paracosm configurations based on combinatorial testing and low dispersion sequences. Paracosm comes with an automatic test case generator that uses random sampling for discrete parameters and deterministic quasi-Monte Carlo generation for continuous parameters. Through an empirical evaluation, we demonstrate the modeling and testing capabilities of Paracosm on a suite of autonomous driving systems implemented using deep neural networks developed in research and education. We show how Paracosm can expose incorrect behaviors or degraded performance.