Cost-effective Simulation-based Test Selection in Self-driving Cars Software

TL;DR

This paper introduces SDCScissor, a machine learning framework that efficiently predicts uninformative tests in self-driving car simulations, significantly reducing testing costs and improving fault detection efficiency.

Contribution

The paper presents a novel ML-based approach to pre-select uninformative tests in simulation, enhancing cost-effectiveness in self-driving car software testing.

Findings

Achieved up to 96% F1-score in test classification.

Outperformed random baseline in detecting more failing tests per time.

Demonstrated effectiveness on large datasets with over 22,000 tests.

Abstract

Simulation environments are essential for the continuous development of complex cyber-physical systems such as self-driving cars (SDCs). Previous results on simulation-based testing for SDCs have shown that many automatically generated tests do not strongly contribute to identification of SDC faults, hence do not contribute towards increasing the quality of SDCs. Because running such "uninformative" tests generally leads to a waste of computational resources and a drastic increase in the testing cost of SDCs, testers should avoid them. However, identifying "uninformative" tests before running them remains an open challenge. Hence, this paper proposes SDCScissor, a framework that leverages Machine Learning (ML) to identify SDC tests that are unlikely to detect faults in the SDC software under test, thus enabling testers to skip their execution and drastically increase the cost-effectiveness of simulation-based testing of SDCs software. Our evaluation concerning the usage of six ML models on two large datasets characterized by 22'652 tests showed that SDC-Scissor achieved a classification F1-score up to 96%. Moreover, our results show that SDC-Scissor outperformed a randomized baseline in identifying more failing tests per time unit. Webpage & Video: https://github.com/ChristianBirchler/sdc-scissor