Training Integer-Only Deep Recurrent Neural Networks

TL;DR

This paper introduces a quantization-aware training method for integer-only RNNs, enabling efficient deployment on edge devices with significant improvements in runtime and model size while maintaining accuracy.

Contribution

The authors develop a novel training approach supporting layer normalization, attention, and PWL activation functions for integer RNNs, improving efficiency without sacrificing accuracy.

Findings

2x faster runtime on edge devices

4x smaller model size

Maintains accuracy comparable to full-precision RNNs

Abstract

Recurrent neural networks (RNN) are the backbone of many text and speech applications. These architectures are typically made up of several computationally complex components such as; non-linear activation functions, normalization, bi-directional dependence and attention. In order to maintain good accuracy, these components are frequently run using full-precision floating-point computation, making them slow, inefficient and difficult to deploy on edge devices. In addition, the complex nature of these operations makes them challenging to quantize using standard quantization methods without a significant performance drop. We present a quantization-aware training method for obtaining a highly accurate integer-only recurrent neural network (iRNN). Our approach supports layer normalization, attention, and an adaptive piecewise linear (PWL) approximation of activation functions, to serve a wide range of state-of-the-art RNNs. The proposed method enables RNN-based language models to run on edge devices with $2\times$ improvement in runtime, and $4\times$ reduction in model size while maintaining similar accuracy as its full-precision counterpart.