Training and Inference with Integers in Deep Neural Networks

TL;DR

This paper introduces WAGE, a novel method for discretizing both training and inference in deep neural networks using low-bitwidth integers, enabling efficient deployment on fixed-point hardware with maintained accuracy.

Contribution

The work presents a new approach to discretize all training and inference processes in neural networks, facilitating deployment on integer-based hardware systems.

Findings

Achieves comparable accuracy with low-bitwidth integer discretization.

Demonstrates potential for hardware deployment with energy efficiency.

Improves regularization through discretization.

Abstract

Researches on deep neural networks with discrete parameters and their deployment in embedded systems have been active and promising topics. Although previous works have successfully reduced precision in inference, transferring both training and inference processes to low-bitwidth integers has not been demonstrated simultaneously. In this work, we develop a new method termed as "WAGE" to discretize both training and inference, where weights (W), activations (A), gradients (G) and errors (E) among layers are shifted and linearly constrained to low-bitwidth integers. To perform pure discrete dataflow for fixed-point devices, we further replace batch normalization by a constant scaling layer and simplify other components that are arduous for integer implementation. Improved accuracies can be obtained on multiple datasets, which indicates that WAGE somehow acts as a type of regularization. Empirically, we demonstrate the potential to deploy training in hardware systems such as integer-based deep learning accelerators and neuromorphic chips with comparable accuracy and higher energy efficiency, which is crucial to future AI applications in variable scenarios with transfer and continual learning demands.