Evaluating the Impact of Flaky Simulators on Testing Autonomous Driving Systems

TL;DR

This paper investigates the prevalence of flaky simulations in testing autonomous driving systems, demonstrating that machine learning can effectively identify flaky tests with fewer reruns, thus improving testing reliability.

Contribution

The study provides empirical evidence of test flakiness in ADS simulators and introduces ML classifiers that detect flaky tests with high accuracy using only one test run.

Findings

Test flakiness is common in ADS simulation testing.

ML classifiers achieve high F1-scores (up to 96%) in identifying flaky tests.

ML-based detection outperforms non-ML baseline requiring multiple test executions.

Abstract

Simulators are widely used to test Autonomous Driving Systems (ADS), but their potential flakiness can lead to inconsistent test results. We investigate test flakiness in simulation-based testing of ADS by addressing two key questions: (1) How do flaky ADS simulations impact automated testing that relies on randomized algorithms? and (2) Can machine learning (ML) effectively identify flaky ADS tests while decreasing the required number of test reruns? Our empirical results, obtained from two widely-used open-source ADS simulators and five diverse ADS test setups, show that test flakiness in ADS is a common occurrence and can significantly impact the test results obtained by randomized algorithms. Further, our ML classifiers effectively identify flaky ADS tests using only a single test run, achieving F1-scores of $85$%, $82$% and $96$% for three different ADS test setups. Our classifiers significantly outperform our non-ML baseline, which requires executing tests at least twice, by $31$%, $21$%, and $13$% in F1-score performance, respectively. We conclude with a discussion on the scope, implications and limitations of our study. We provide our complete replication package in a Github repository.