Two Sparsities Are Better Than One: Unlocking the Performance Benefits of Sparse-Sparse Networks

TL;DR

This paper introduces Complementary Sparsity, a technique that leverages both weight and activation sparsity in neural networks to achieve up to 100X improvements in throughput and energy efficiency on FPGAs.

Contribution

The paper proposes a novel method called Complementary Sparsity that effectively combines weight and activation sparsity to enhance neural network performance on existing hardware.

Findings

Achieved up to 100X throughput and energy efficiency improvements.

Demonstrated scalability and resource tradeoffs for convolutional networks.

Showed potential for efficient scaling of future AI models.

Abstract

In principle, sparse neural networks should be significantly more efficient than traditional dense networks. Neurons in the brain exhibit two types of sparsity; they are sparsely interconnected and sparsely active. These two types of sparsity, called weight sparsity and activation sparsity, when combined, offer the potential to reduce the computational cost of neural networks by two orders of magnitude. Despite this potential, today's neural networks deliver only modest performance benefits using just weight sparsity, because traditional computing hardware cannot efficiently process sparse networks. In this article we introduce Complementary Sparsity, a novel technique that significantly improves the performance of dual sparse networks on existing hardware. We demonstrate that we can achieve high performance running weight-sparse networks, and we can multiply those speedups by incorporating activation sparsity. Using Complementary Sparsity, we show up to 100X improvement in throughput and energy efficiency performing inference on FPGAs. We analyze scalability and resource tradeoffs for a variety of kernels typical of commercial convolutional networks such as ResNet-50 and MobileNetV2. Our results with Complementary Sparsity suggest that weight plus activation sparsity can be a potent combination for efficiently scaling future AI models.