Delta Networks for Optimized Recurrent Network Computation

TL;DR

This paper introduces delta networks, a novel RNN architecture that transmits neuron values only when significant changes occur, leading to substantial reductions in computation and memory access with minimal accuracy loss.

Contribution

The paper proposes delta networks for RNNs, optimizing their execution by transmitting only significant activation changes, and demonstrates substantial speedups on multiple benchmarks.

Findings

9X reduction in cost for TIDIGITS with negligible accuracy loss

5.7X speedup on Wall Street Journal speech recognition

100X reduction in RNN cost for driving angle prediction

Abstract

Many neural networks exhibit stability in their activation patterns over time in response to inputs from sensors operating under real-world conditions. By capitalizing on this property of natural signals, we propose a Recurrent Neural Network (RNN) architecture called a delta network in which each neuron transmits its value only when the change in its activation exceeds a threshold. The execution of RNNs as delta networks is attractive because their states must be stored and fetched at every timestep, unlike in convolutional neural networks (CNNs). We show that a naive run-time delta network implementation offers modest improvements on the number of memory accesses and computes, but optimized training techniques confer higher accuracy at higher speedup. With these optimizations, we demonstrate a 9X reduction in cost with negligible loss of accuracy for the TIDIGITS audio digit recognition benchmark. Similarly, on the large Wall Street Journal speech recognition benchmark even existing networks can be greatly accelerated as delta networks, and a 5.7x improvement with negligible loss of accuracy can be obtained through training. Finally, on an end-to-end CNN trained for steering angle prediction in a driving dataset, the RNN cost can be reduced by a substantial 100X.