Machine Learning-based Test Selection for Simulation-based Testing of Self-driving Cars Software

TL;DR

This paper introduces SDC-Scissor, a machine learning-based test selection method that enhances simulation testing efficiency for self-driving cars by accurately predicting and skipping uninformative tests, reducing testing time and increasing fault detection.

Contribution

The paper presents a novel ML-driven approach, SDC-Scissor, for selecting valuable tests in simulation-based self-driving car testing, significantly improving cost-effectiveness and fault detection without added computational overhead.

Findings

Achieved up to 93.4% classification accuracy in fault prediction.

Reduced testing time by approximately 170%.

Detected 33% more failure-triggering tests than random selection.

Abstract

Abstract Simulation platforms facilitate the development of emerging cyber-physical systems (CPS) like self-driving cars (SDC) because they are more efficient and less dangerous than field operational tests. Despite this, thoroughly testing SDCs in simulated environments remains challenging because SDCs must be tested in a sheer amount of long-running test scenarios. Past results on software testing optimization have shown that not all the tests contribute equally to establishing confidence in test subjects' quality and reliability, with some \uninformative" tests that can be skipped (or removed) to reduce testing effort. However, this problem was partially addressed in the context of SDC simulation platforms. In this paper, we investigate test selection strategies to increase the cost-effectiveness of simulation-based testing in the context of SDCs. We propose an approach called SDC-Scissor (SDC coSt-effeCtIve teSt SelectOR), which leverages machine learning (ML) strategies to identify and skip tests that are unlikely to detect faults in SDCs before executing them. Specifically, SDC-Scissor extract features concerning the characteristics of the test scenarios being executed in the simulation environment and via ML strategies predict tests that lead to faults before executing them. Our evaluation shows that SDC-Scissor achieved high classification accuracy (up to 93.4%) in classifying tests leading to a fault which allows improving testing cost-effectiveness: SDC-Scissor was able to reduce (ca. 170%) the time spent in running irrelevant tests as well as identified 33% more failure triggering tests compared to a randomized baseline. Interestingly, SDC-Scissor does not introduce significant computational overhead in the SDCs testing process, which is critical to SDC development in industrial settings.