Mixture-of-Rookies: Saving DNN Computations by Predicting ReLU Outputs

TL;DR

This paper introduces Mixture-of-Rookies, a predictor that efficiently forecasts zero outputs of ReLU neurons to skip unnecessary computations, thereby accelerating DNN inference and reducing energy use.

Contribution

It presents a novel hybrid predictor combining linear correlation and clustering based on angles to predict ReLU outputs, improving DNN efficiency.

Findings

Achieves 1.2x speedup in DNN inference

Reduces energy consumption by 16.5% on average

Adds only 5.3% area overhead

Abstract

Deep Neural Networks (DNNs) are widely used in many applications domains. However, they require a vast amount of computations and memory accesses to deliver outstanding accuracy. In this paper, we propose a scheme to predict whether the output of each ReLu activated neuron will be a zero or a positive number in order to skip the computation of those neurons that will likely output a zero. Our predictor, named Mixture-of-Rookies, combines two inexpensive components. The first one exploits the high linear correlation between binarized (1-bit) and full-precision (8-bit) dot products, whereas the second component clusters together neurons that tend to output zero at the same time. We propose a novel clustering scheme based on the analysis of angles, as the sign of the dot product of two vectors depends on the cosine of the angle between them. We implement our hybrid zero output predictor on top of a state-of-the-art DNN accelerator. Experimental results show that our scheme introduces a small area overhead of 5.3% while achieving a speedup of 1.2x and reducing energy consumption by 16.5% on average for a set of diverse DNNs.