NITRO-D: Native Integer-only Training of Deep Convolutional Neural Networks

TL;DR

NITRO-D introduces a novel framework for training deep convolutional neural networks entirely in the integer domain, eliminating the need for floating-point operations during training and inference, thus enabling more efficient deployment in resource-constrained environments.

Contribution

It presents the first comprehensive integer-only training method for deep CNNs, including new architecture components and an optimizer tailored for integer computations.

Findings

Improves accuracy of integer MLPs by up to 5.96% over state-of-the-art.

Reduces memory and energy consumption by up to 76.14% and 32.42%.

Successfully trains integer-only CNNs with competitive performance.

Abstract

Quantization is a pivotal technique for managing the growing computational and memory demands of Deep Neural Networks (DNNs). By reducing the number of bits used to represent weights and activations (typically from 32-bit Floating-Point (FP) to 16-bit or 8-bit integers), quantization reduces memory footprint, energy consumption, and execution time of DNNs. However, most existing methods typically target DNN inference, while training still relies on FP operations, limiting applicability in environments where FP arithmetic is unavailable. To date, only one prior work has addressed integer-only training, and only for Multi-Layer Perceptron (MLP) architectures. This paper introduces NITRO-D, a novel framework for training deep integer-only Convolutional Neural Networks (CNNs) that operate entirely in the integer domain for both training and inference. NITRO-D enables training of integer CNNs without requiring a separate quantization scheme. Specifically, it introduces a novel architecture that integrates multiple local-loss blocks, which include the proposed NITRO-Scaling layer and NITRO-ReLU activation function. The proposed framework also features a novel learning algorithm that employs local error signals and leverages IntegerSGD, an optimizer specifically designed for integer computations. NITRO-D is implemented as an open-source Python library. Extensive evaluations on state-of-the-art image recognition datasets demonstrate its effectiveness. For integer-only MLPs, NITRO-D improves test accuracy by up to +5.96% over the state-of-the-art. It also successfully trains integer-only CNNs, reducing memory requirements and energy consumption by up to 76.14% and 32.42%, respectively, compared to the traditional FP backpropagation algorithm.