Evaluating Roadside Perception for Autonomous Vehicles: Insights from Field Testing

TL;DR

This paper presents a new evaluation methodology for roadside perception systems in autonomous vehicles, validated through real-world testing at Mcity, enabling better assessment and comparison of system performance.

Contribution

It introduces a comprehensive, real-world evaluation framework for roadside perception systems, addressing the lack of standardized benchmarks in this emerging field.

Findings

Comparative analysis of perception systems in realistic scenarios

Identification of strengths and limitations of existing systems

Insights to inform industry standards and future research

Abstract

Roadside perception systems are increasingly crucial in enhancing traffic safety and facilitating cooperative driving for autonomous vehicles. Despite rapid technological advancements, a major challenge persists for this newly arising field: the absence of standardized evaluation methods and benchmarks for these systems. This limitation hampers the ability to effectively assess and compare the performance of different systems, thus constraining progress in this vital field. This paper introduces a comprehensive evaluation methodology specifically designed to assess the performance of roadside perception systems. Our methodology encompasses measurement techniques, metric selection, and experimental trial design, all grounded in real-world field testing to ensure the practical applicability of our approach. We applied our methodology in Mcity\footnote{\url{https://mcity.umich.edu/}}, a controlled testing environment, to evaluate various off-the-shelf perception systems. This approach allowed for an in-depth comparative analysis of their performance in realistic scenarios, offering key insights into their respective strengths and limitations. The findings of this study are poised to inform the development of industry-standard benchmarks and evaluation methods, thereby enhancing the effectiveness of roadside perception system development and deployment for autonomous vehicles. We anticipate that this paper will stimulate essential discourse on standardizing evaluation methods for roadside perception systems, thus pushing the frontiers of this technology. Furthermore, our results offer both academia and industry a comprehensive understanding of the capabilities of contemporary infrastructure-based perception systems.