Testing Autonomous Systems with Believed Equivalence Refinement

TL;DR

This paper introduces believed equivalence, a method for refining test case categories in autonomous systems based on expert belief and test results, improving testing accuracy for deep neural network modules.

Contribution

It proposes a novel believed equivalence refinement approach that dynamically updates equivalence classes during autonomous system testing, especially for neural network modules.

Findings

Effective refinement of equivalence classes demonstrated on autonomous driving modules

Analytical and lazy methods for equivalence refinement proposed

Potential to enhance testing coverage and system reliability

Abstract

Continuous engineering of autonomous driving functions commonly requires deploying vehicles in road testing to obtain inputs that cause problematic decisions. Although the discovery leads to producing an improved system, it also challenges the foundation of testing using equivalence classes and the associated relative test coverage criterion. In this paper, we propose believed equivalence, where the establishment of an equivalence class is initially based on expert belief and is subject to a set of available test cases having a consistent valuation. Upon a newly encountered test case that breaks the consistency, one may need to refine the established categorization in order to split the originally believed equivalence into two. Finally, we focus on modules implemented using deep neural networks where every category partitions an input over the real domain. We present both analytical and lazy methods to suggest the refinement. The concept is demonstrated in analyzing multiple autonomous driving modules, indicating the potential of our proposed approach.