MLPerf Automotive

TL;DR

MLPerf Automotive introduces a standardized benchmark for evaluating AI systems in automotive applications, focusing on safety-critical, real-time perception tasks to enable consistent performance comparisons across hardware and software.

Contribution

It is the first benchmark specifically designed for automotive machine learning systems, addressing unique safety and real-time constraints with standardized evaluation protocols.

Findings

First benchmark for automotive AI systems

Includes perception tasks like 2D/3D object detection and segmentation

Provides reproducible performance evaluation methods

Abstract

We present MLPerf Automotive, the first standardized public benchmark for evaluating Machine Learning systems that are deployed for AI acceleration in automotive systems. Developed through a collaborative partnership between MLCommons and the Autonomous Vehicle Computing Consortium, this benchmark addresses the need for standardized performance evaluation methodologies in automotive machine learning systems. Existing benchmark suites cannot be utilized for these systems since automotive workloads have unique constraints including safety and real-time processing that distinguish them from the domains that previously introduced benchmarks target. Our benchmarking framework provides latency and accuracy metrics along with evaluation protocols that enable consistent and reproducible performance comparisons across different hardware platforms and software implementations. The first iteration of the benchmark consists of automotive perception tasks in 2D object detection, 2D semantic segmentation, and 3D object detection. We describe the methodology behind the benchmark design including the task selection, reference models, and submission rules. We also discuss the first round of benchmark submissions and the challenges involved in acquiring the datasets and the engineering efforts to develop the reference implementations. Our benchmark code is available at https://github.com/mlcommons/mlperf_automotive.