Using Cooperative Co-evolutionary Search to Generate Metamorphic Test Cases for Autonomous Driving Systems

TL;DR

This paper presents CoCoMEGA, a novel automated testing framework that combines metamorphic testing with cooperative co-evolutionary algorithms to generate diverse, safety-relevant test cases for autonomous driving systems in simulation.

Contribution

It introduces CoCoMEGA, a new approach integrating metamorphic testing with search algorithms to improve safety assessment of autonomous driving systems.

Findings

Outperforms baseline methods in generating safety-critical test cases

Achieves broader exploration of test scenarios in simulation

Demonstrates effectiveness and efficiency in identifying undesirable behaviors

Abstract

Autonomous Driving Systems (ADSs) rely on Deep Neural Networks, allowing vehicles to navigate complex, open environments. However, the unpredictability of these scenarios highlights the need for rigorous system-level testing to ensure safety, a task usually performed with a simulator in the loop. Though one important goal of such testing is to detect safety violations, there are many undesirable system behaviors, that may not immediately lead to violations, that testing should also be focusing on, thus detecting more subtle problems and enabling a finer-grained analysis. This paper introduces Cooperative Co-evolutionary MEtamorphic test Generator for Autonomous systems (CoCoMEGA), a novel automated testing framework aimed at advancing system-level safety assessments of ADSs. CoCoMEGA combines Metamorphic Testing (MT) with a search-based approach utilizing Cooperative Co-Evolutionary Algorithms (CCEA) to efficiently generate a diverse set of test cases. CoCoMEGA emphasizes the identification of test scenarios that present undesirable system behavior, that may eventually lead to safety violations, captured by Metamorphic Relations (MRs). When evaluated within the CARLA simulation environment on the Interfuser ADS, CoCoMEGA consistently outperforms baseline methods, demonstrating enhanced effectiveness and efficiency in generating severe, diverse MR violations and achieving broader exploration of the test space. These results underscore CoCoMEGA as a promising, more scalable solution to the inherent challenges in ADS testing with a simulator in the loop. Future research directions may include extending the approach to additional simulation platforms, applying it to other complex systems, and exploring methods for further improving testing efficiency such as surrogate modeling.