A Language for Autonomous Vehicles Testing Oracles

TL;DR

This paper introduces a domain-specific language for defining autonomous vehicle testing oracles independently of simulation tools, enabling more consistent and expressive safety and performance evaluations.

Contribution

It presents a new language for AV testing oracles that are decoupled from specific simulators, allowing flexible and consistent evaluation of AV behaviors.

Findings

Existing oracles evaluate AVs inconsistently.

The language can encode safety, liveness, and temporal properties.

Different oracles can rank the same AV solutions differently.

Abstract

Testing autonomous vehicles (AVs) requires complex oracles to determine if the AVs behavior conforms with specifications and humans' expectations. Available open source oracles are tightly embedded in the AV simulation software and are developed and implemented in an ad hoc way. We propose a domain specific language that enables defining oracles independent of the AV solutions and the simulator. A testing analyst can encode safety, liveness, timeliness and temporal properties in our language. To show the expressiveness of our language we implement three different types of available oracles. We find that the same AV solutions may be ranked significantly differently across existing oracles, thus existing oracles do not evaluate AVs in a consistent manner.