Recurrent Neural Networks With Limited Numerical Precision

TL;DR

This paper explores reducing the numerical precision of RNNs through various quantization methods, demonstrating that low-precision RNNs can maintain or improve accuracy, enabling more efficient hardware implementations.

Contribution

It introduces and evaluates stochastic and deterministic low-precision training methods for RNNs, showing their effectiveness across multiple datasets.

Findings

Low-precision RNNs achieve comparable or higher accuracy.

Quantization methods like ternarization and pow2-ternarization are effective.

Results support hardware-efficient RNN implementations.

Abstract

Recurrent Neural Networks (RNNs) produce state-of-art performance on many machine learning tasks but their demand on resources in terms of memory and computational power are often high. Therefore, there is a great interest in optimizing the computations performed with these models especially when considering development of specialized low-power hardware for deep networks. One way of reducing the computational needs is to limit the numerical precision of the network weights and biases, and this will be addressed for the case of RNNs. We present results from the use of different stochastic and deterministic reduced precision training methods applied to two major RNN types, which are then tested on three datasets. The results show that the stochastic and deterministic ternarization, pow2- ternarization, and exponential quantization methods gave rise to low-precision RNNs that produce similar and even higher accuracy on certain datasets, therefore providing a path towards training more efficient implementations of RNNs in specialized hardware.