Disengagement Analysis and Field Tests of a Prototypical Open-Source Level 4 Autonomous Driving System

TL;DR

This paper evaluates an open-source Level 4 autonomous vehicle over 236 km, analyzing disengagements to identify robustness issues and areas needing improvement for safer deployment.

Contribution

It introduces a novel five-level disengagement criticality framework and provides real-world insights into the performance of open-source autonomous driving systems.

Findings

Disengagement rate of 0.127 per km observed.

Perception failures account for 40% of disengagements.

Frequent interventions indicate trust issues with the system.

Abstract

Proprietary Autonomous Driving Systems are typically evaluated through disengagements, unplanned manual interventions to alter vehicle behavior, as annually reported by the California Department of Motor Vehicles. However, the real-world capabilities of prototypical open-source Level 4 vehicles over substantial distances remain largely unexplored. This study evaluates a research vehicle running an Autoware-based software stack across 236 km of mixed traffic. By classifying 30 disengagements across 26 rides with a novel five-level criticality framework, we observed a spatial disengagement rate of 0.127 1/km. Interventions predominantly occurred at lower speeds near static objects and traffic lights. Perception and Planning failures accounted for 40% and 26.7% of disengagements, respectively, largely due to object-tracking losses and operational deadlocks caused by parked vehicles. Frequent, unnecessary interventions highlighted a lack of trust on the part of the safety driver. These results show that while open-source software enables extensive operations, disengagement analysis is vital for uncovering robustness issues missed by standard metrics.