Benchmarking Contemporary Deep Learning Hardware and Frameworks:A Survey of Qualitative Metrics

TL;DR

This survey reviews benchmarking principles, hardware devices, and software frameworks for deep learning, analyzing their characteristics and metrics, and discusses MLPerf benchmark results to evaluate performance across platforms.

Contribution

It provides a comprehensive overview of benchmarking principles, compares deep learning hardware and frameworks qualitatively, and summarizes industry-standard MLPerf benchmark results.

Findings

MLPerf benchmarks evaluate training and inference performance.

Mainstream AI devices differ in key qualitative characteristics.

Benchmarking principles guide fair and effective comparison.

Abstract

This paper surveys benchmarking principles, machine learning devices including GPUs, FPGAs, and ASICs, and deep learning software frameworks. It also reviews these technologies with respect to benchmarking from the perspectives of a 6-metric approach to frameworks and an 11-metric approach to hardware platforms. Because MLPerf is a benchmark organization working with industry and academia, and offering deep learning benchmarks that evaluate training and inference on deep learning hardware devices, the survey also mentions MLPerf benchmark results, benchmark metrics, datasets, deep learning frameworks and algorithms. We summarize seven benchmarking principles, differential characteristics of mainstream AI devices, and qualitative comparison of deep learning hardware and frameworks.