Fine-grained Testing for Autonomous Driving Software: a Study on Autoware with LLM-driven Unit Testing

TL;DR

This paper investigates fine-grained unit testing for autonomous driving software, specifically Autoware, comparing human-written and LLM-generated test cases, and proposes a new approach to improve test coverage and reliability.

Contribution

It is the first study on unit testing for ADS source code, introducing AwTest-LLM to enhance test coverage and address challenges with LLM-generated tests.

Findings

Human-written tests have limited coverage in Autoware.

LLM-generated tests face significant challenges in application.

AwTest-LLM improves test coverage and pass rates.

Abstract

Testing autonomous driving systems (ADS) is critical to ensuring their reliability and safety. Existing ADS testing works focuses on designing scenarios to evaluate system-level behaviors, while fine-grained testing of ADS source code has received comparatively little attention. To address this gap, we present the first study on testing, specifically unit testing, for ADS source code. Our study focuses on an industrial ADS framework, Autoware. We analyze both human-written test cases and those generated by large language models (LLMs). Our findings reveal that human-written test cases in Autoware exhibit limited test coverage, and significant challenges remain in applying LLM-generated tests for Autoware unit testing. To overcome these challenges, we propose AwTest-LLM, a novel approach to enhance test coverage and improve test case pass rates across Autoware packages.