WRPN: Wide Reduced-Precision Networks

TL;DR

WRPN introduces a method to train neural networks with reduced-precision activations and increased filter maps, achieving comparable or better accuracy than full-precision models while improving computational efficiency.

Contribution

The paper proposes a novel scheme to use reduced-precision activations combined with wider networks, maintaining accuracy and enhancing efficiency in deep neural networks.

Findings

WRPN matches or surpasses baseline accuracy on ILSVRC-12.

The scheme reduces memory footprint and computational energy.

WRPN outperforms previous reduced-precision networks in accuracy and efficiency.

Abstract

For computer vision applications, prior works have shown the efficacy of reducing numeric precision of model parameters (network weights) in deep neural networks. Activation maps, however, occupy a large memory footprint during both the training and inference step when using mini-batches of inputs. One way to reduce this large memory footprint is to reduce the precision of activations. However, past works have shown that reducing the precision of activations hurts model accuracy. We study schemes to train networks from scratch using reduced-precision activations without hurting accuracy. We reduce the precision of activation maps (along with model parameters) and increase the number of filter maps in a layer, and find that this scheme matches or surpasses the accuracy of the baseline full-precision network. As a result, one can significantly improve the execution efficiency (e.g. reduce dynamic memory footprint, memory bandwidth and computational energy) and speed up the training and inference process with appropriate hardware support. We call our scheme WRPN - wide reduced-precision networks. We report results and show that WRPN scheme is better than previously reported accuracies on ILSVRC-12 dataset while being computationally less expensive compared to previously reported reduced-precision networks.