It's all about PR -- Smart Benchmarking AI Accelerators using Performance Representatives

TL;DR

This paper introduces a performance modeling method for AI hardware accelerators that uses Performance Representatives to reduce training data needs while maintaining high accuracy in performance estimation.

Contribution

The paper presents a novel approach leveraging hardware architecture knowledge and initial parameter sweeps to identify PRs, significantly reducing training samples for performance modeling.

Findings

Achieved MAPE of 0.02% for single-layer estimations

Achieved MAPE of 0.68% for whole DNN estimations

Reduced training samples to less than 10,000 while maintaining accuracy

Abstract

Statistical models are widely used to estimate the performance of commercial off-the-shelf (COTS) AI hardware accelerators. However, training of statistical performance models often requires vast amounts of data, leading to a significant time investment and can be difficult in case of limited hardware availability. To alleviate this problem, we propose a novel performance modeling methodology that significantly reduces the number of training samples while maintaining good accuracy. Our approach leverages knowledge of the target hardware architecture and initial parameter sweeps to identify a set of Performance Representatives (PR) for deep neural network (DNN) layers. These PRs are then used for benchmarking, building a statistical performance model, and making estimations. This targeted approach drastically reduces the number of training samples needed, opposed to random sampling, to achieve a better estimation accuracy. We achieve a Mean Absolute Percentage Error (MAPE) of as low as 0.02% for single-layer estimations and 0.68% for whole DNN estimations with less than 10000 training samples. The results demonstrate the superiority of our method for single-layer estimations compared to models trained with randomly sampled datasets of the same size.