TensorDash: Exploiting Sparsity to Accelerate Deep Neural Network Training and Inference

TL;DR

TensorDash is a hardware technique that leverages sparsity in neural network data to significantly accelerate training and improve energy efficiency in deep learning accelerators.

Contribution

TensorDash introduces a hardware-level method combining a sparse operand interconnect and scheduler to exploit sparsity for faster, more energy-efficient neural network training.

Findings

Accelerates training by 1.95x across various models

Increases energy efficiency by 1.89x

Effective with multiple data types, including float and bfloat16

Abstract

TensorDash is a hardware level technique for enabling data-parallel MAC units to take advantage of sparsity in their input operand streams. When used to compose a hardware accelerator for deep learning, TensorDash can speedup the training process while also increasing energy efficiency. TensorDash combines a low-cost, sparse input operand interconnect comprising an 8-input multiplexer per multiplier input, with an area-efficient hardware scheduler. While the interconnect allows a very limited set of movements per operand, the scheduler can effectively extract sparsity when it is present in the activations, weights or gradients of neural networks. Over a wide set of models covering various applications, TensorDash accelerates the training process by $1.95{\times}$ while being $1.89\times$ more energy-efficient, $1.6\times$ more energy efficient when taking on-chip and off-chip memory accesses into account. While TensorDash works with any datatype, we demonstrate it with both single-precision floating-point units and bfloat16.