Multi-modal Traffic Scenario Generation for Autonomous Driving System Testing

TL;DR

This paper introduces TrafficComposer, a multi-modal approach combining natural language and images to generate realistic traffic scenarios for autonomous driving system testing, improving scenario quality and bug detection efficiency.

Contribution

TrafficComposer is the first method to integrate NL descriptions and scene images for high-quality traffic scenario generation in ADS testing.

Findings

Achieves 97.0% accuracy on 120 scenarios, outperforming baselines by 7.3%.

Generates scenarios that discover 37 bugs in ADS testing.

Enhances fuzz testing by 33-124% in bug detection.

Abstract

Autonomous driving systems (ADS) require extensive testing and validation before deployment. However, it is tedious and time-consuming to construct traffic scenarios for ADS testing. In this paper, we propose TrafficComposer, a multi-modal traffic scenario construction approach for ADS testing. TrafficComposer takes as input a natural language (NL) description of a desired traffic scenario and a complementary traffic scene image. Then, it generates the corresponding traffic scenario in a simulator, such as CARLA and LGSVL. Specifically, TrafficComposer integrates high-level dynamic information about the traffic scenario from the NL description and intricate details about the surrounding vehicles, pedestrians, and the road network from the image. The information from the two modalities is complementary to each other and helps generate high-quality traffic scenarios for ADS testing. On a benchmark of 120 traffic scenarios, TrafficComposer achieves 97.0% accuracy, outperforming the best-performing baseline by 7.3%. Both direct testing and fuzz testing experiments on six ADSs prove the bug detection capabilities of the traffic scenarios generated by TrafficComposer. These scenarios can directly discover 37 bugs and help two fuzzing methods find 33%--124% more bugs serving as initial seeds.