Demystifying the MLPerf Benchmark Suite

TL;DR

This paper analyzes the MLPerf benchmark suite, highlighting its diversity, system bottlenecks, and the impact of hardware and software optimizations on deep learning performance.

Contribution

It provides a detailed characterization of MLPerf, compares it with previous benchmarks, and offers insights into system bottlenecks and optimization strategies for deep learning.

Findings

Dedicated GPU interconnects improve distributed training.

Scaling efficiency varies across MLPerf models.

Compiler optimizations and mixed-precision training enhance performance.

Abstract

MLPerf, an emerging machine learning benchmark suite strives to cover a broad range of applications of machine learning. We present a study on its characteristics and how the MLPerf benchmarks differ from some of the previous deep learning benchmarks like DAWNBench and DeepBench. We find that application benchmarks such as MLPerf (although rich in kernels) exhibit different features compared to kernel benchmarks such as DeepBench. MLPerf benchmark suite contains a diverse set of models which allows unveiling various bottlenecks in the system. Based on our findings, dedicated low latency interconnect between GPUs in multi-GPU systems is required for optimal distributed deep learning training. We also observe variation in scaling efficiency across the MLPerf models. The variation exhibited by the different models highlight the importance of smart scheduling strategies for multi-GPU training. Another observation is that CPU utilization increases with increase in number of GPUs used for training. Corroborating prior work we also observe and quantify improvements possible by compiler optimizations, mixed-precision training and use of Tensor Cores.