scenoRITA: Generating Less-Redundant, Safety-Critical and Motion Sickness-Inducing Scenarios for Autonomous Vehicles

TL;DR

scenoRITA is a novel evolutionary test generation approach for autonomous vehicles that creates diverse, valid, and safety-critical scenarios, significantly improving violation detection over existing methods.

Contribution

It introduces a new gene representation, multiple test oracles, and a duplicate elimination technique to enhance scenario generation for AV testing.

Findings

Generated 1,026 unique violations, outperforming random and state-of-the-art methods.

Increased violation detection by approximately 23-24%.

Produced more effective and diverse safety-critical scenarios.

Abstract

There is tremendous global enthusiasm for research, development, and deployment of autonomous vehicles (AVs), e.g., self-driving taxis and trucks from Waymo and Baidu. The current practice for testing AVs uses virtual tests-where AVs are tested in software simulations-since they offer a more efficient and safer alternative compared to field operational tests. Specifically, search-based approaches are used to find particularly critical situations. These approaches provide an opportunity to automatically generate tests; however, systematically creating valid and effective tests for AV software remains a major challenge. To address this challenge, we introduce scenoRITA, a test generation approach for AVs that uses evolutionary algorithms with (1) a novel gene representation that allows obstacles to be fully mutable, hence, resulting in more reported violations, (2) 5 test oracles to determine both safety and motion sickness-inducing violations, and (3) a novel technique to identify and eliminate duplicate tests. Our extensive evaluation shows that scenoRITA can produce effective driving scenarios that expose an ego car to safety critical situations. scenoRITA generated tests that resulted in a total of 1,026 unique violations, increasing the number of reported violations by 23.47% and 24.21% compared to random test generation and state-of-the-art partially-mutable test generation, respectively.