ComOpT: Combination and Optimization for Testing Autonomous Driving Systems

TL;DR

ComOpT is an open-source tool that systematically generates diverse, constraint-feasible scenarios to test autonomous driving systems, effectively uncovering safety and performance issues with limited testing resources.

Contribution

It introduces a novel combination of combinatorial scenario generation, constraint handling, and scenario instantiation for autonomous vehicle testing, demonstrated on Apollo 6.

Findings

Generated diverse scenarios with limited test budgets

Uncovered safety-critical situations like failed turns and dangerous behaviors

Won first place in the 2021 IEEE AV Testing Challenge

Abstract

ComOpT is an open-source research tool for coverage-driven testing of autonomous driving systems, focusing on planning and control. Starting with (i) a meta-model characterizing discrete conditions to be considered and (ii) constraints specifying the impossibility of certain combinations, ComOpT first generates constraint-feasible abstract scenarios while maximally increasing the coverage of k-way combinatorial testing. Each abstract scenario can be viewed as a conceptual equivalence class, which is then instantiated into multiple concrete scenarios by (1) randomly picking one local map that fulfills the specified geographical condition, and (2) assigning all actors accordingly with parameters within the range. Finally, ComOpT evaluates each concrete scenario against a set of KPIs and performs local scenario variation via spawning a new agent that might lead to a collision at designated points. We use ComOpT to test the Apollo~6 autonomous driving software stack. ComOpT can generate highly diversified scenarios with limited test budgets while uncovering problematic situations such as inabilities to make simple right turns, uncomfortable accelerations, and dangerous driving patterns. ComOpT participated in the 2021 IEEE AI Autonomous Vehicle Testing Challenge and won first place among more than 110 contending teams.