MLPerf Training Benchmark

TL;DR

MLPerf is a comprehensive benchmarking suite designed to evaluate and compare the performance of machine learning training across diverse hardware and software systems, addressing unique challenges like stochasticity and diversity.

Contribution

This paper introduces MLPerf, a standardized benchmark for ML training that overcomes key challenges such as variability, fairness, and diversity in hardware and software environments.

Findings

MLPerf effectively drives performance improvements across vendors.

Benchmark results show high variability in training times, highlighting the need for standardized metrics.

MLPerf facilitates fair comparison of ML training solutions.

Abstract

Machine learning (ML) needs industry-standard performance benchmarks to support design and competitive evaluation of the many emerging software and hardware solutions for ML. But ML training presents three unique benchmarking challenges absent from other domains: optimizations that improve training throughput can increase the time to solution, training is stochastic and time to solution exhibits high variance, and software and hardware systems are so diverse that fair benchmarking with the same binary, code, and even hyperparameters is difficult. We therefore present MLPerf, an ML benchmark that overcomes these challenges. Our analysis quantitatively evaluates MLPerf's efficacy at driving performance and scalability improvements across two rounds of results from multiple vendors.