Introducing Milabench: Benchmarking Accelerators for AI

TL;DR

Milabench is a comprehensive, open-source benchmarking suite designed specifically for AI accelerators, capturing the unique workloads of deep learning applications and aiding procurement decisions.

Contribution

The paper introduces Milabench, a novel benchmarking suite tailored for AI workloads, developed through extensive literature review and community surveys, filling a gap in HPC benchmarking for AI.

Findings

Performance evaluation on NVIDIA, AMD, and Intel GPUs.

Benchmark suite covers diverse AI workloads.

Open source availability for community use.

Abstract

AI workloads, particularly those driven by deep learning, are introducing novel usage patterns to high-performance computing (HPC) systems that are not comprehensively captured by standard HPC benchmarks. As one of the largest academic research centers dedicated to deep learning, Mila identified the need to develop a custom benchmarking suite to address the diverse requirements of its community, which consists of over 1,000 researchers. This report introduces Milabench, the resulting benchmarking suite. Its design was informed by an extensive literature review encompassing 867 papers, as well as surveys conducted with Mila researchers. This rigorous process led to the selection of 26 primary benchmarks tailored for procurement evaluations, alongside 16 optional benchmarks for in-depth analysis. We detail the design methodology, the structure of the benchmarking suite, and provide performance evaluations using GPUs from NVIDIA, AMD, and Intel. The Milabench suite is open source and can be accessed at github.com/mila-iqia/milabench.