The Sparsity Roofline: Understanding the Hardware Limits of Sparse Neural Networks

TL;DR

The paper introduces the Sparsity Roofline, a visual performance model that predicts the hardware performance limits of sparse neural networks by jointly modeling accuracy, sparsity, and speedup without needing optimized kernels.

Contribution

It presents a novel analytical model for predicting sparse network performance and validates it across various architectures and sparsity patterns, aiding researchers and hardware designers.

Findings

The model accurately predicts speedup for different sparsity patterns.

It helps identify sparsity regimes with the highest performance potential.

The approach does not require implementing optimized kernels.

Abstract

We introduce the Sparsity Roofline, a visual performance model for evaluating sparsity in neural networks. The Sparsity Roofline jointly models network accuracy, sparsity, and theoretical inference speedup. Our approach does not require implementing and benchmarking optimized kernels, and the theoretical speedup becomes equal to the actual speedup when the corresponding dense and sparse kernels are well-optimized. We achieve this through a novel analytical model for predicting sparse network performance, and validate the predicted speedup using several real-world computer vision architectures pruned across a range of sparsity patterns and degrees. We demonstrate the utility and ease-of-use of our model through two case studies: (1) we show how machine learning researchers can predict the performance of unimplemented or unoptimized block-structured sparsity patterns, and (2) we show how hardware designers can predict the performance implications of new sparsity patterns and sparse data formats in hardware. In both scenarios, the Sparsity Roofline helps performance experts identify sparsity regimes with the highest performance potential.