Autonomous Vehicle Benchmarking using Unbiased Metrics

TL;DR

This paper introduces a comprehensive set of unbiased metrics for benchmarking autonomous vehicle systems across various scenarios, aiming to better understand their performance, limitations, and generalizability beyond traditional disengagement reports.

Contribution

The paper proposes a new, complete set of unbiased metrics for autonomous vehicle benchmarking, extending evaluation beyond disengagement reports to include diverse performance aspects.

Findings

Metrics applied to UC San Diego's autonomous vehicles during early deployments.

Benchmarking revealed system strengths and weaknesses across different scenarios.

Metrics enable comparison with human drivers and other autonomous systems.

Abstract

With the recent development of autonomous vehicle technology, there have been active efforts on the deployment of this technology at different scales that include urban and highway driving. While many of the prototypes showcased have been shown to operate under specific cases, little effort has been made to better understand their shortcomings and generalizability to new areas. Distance, uptime and number of manual disengagements performed during autonomous driving provide a high-level idea on the performance of an autonomous system but without proper data normalization, testing location information, and the number of vehicles involved in testing, the disengagement reports alone do not fully encompass system performance and robustness. Thus, in this study a complete set of metrics are applied for benchmarking autonomous vehicle systems in a variety of scenarios that can be extended for comparison with human drivers and other autonomous vehicle systems. These metrics have been used to benchmark UC San Diego's autonomous vehicle platforms during early deployments for micro-transit and autonomous mail delivery applications.