Rounding Methods for Neural Networks with Low Resolution Synaptic Weights

TL;DR

This paper introduces two advanced methods for converting high-resolution neural network weights to low-resolution or binary weights, significantly improving performance over standard rounding, and explores their application in hardware-constrained systems.

Contribution

The paper proposes novel mapping techniques for low-resolution weights in neural networks, surpassing standard rounding performance and enabling efficient hardware implementations.

Findings

Performance with advanced methods is substantially better than standard rounding.

Low-resolution weights can still achieve competitive accuracy.

Applicable to hardware systems with very limited memory and resolution.

Abstract

Neural network algorithms simulated on standard computing platforms typically make use of high resolution weights, with floating-point notation. However, for dedicated hardware implementations of such algorithms, fixed-point synaptic weights with low resolution are preferable. The basic approach of reducing the resolution of the weights in these algorithms by standard rounding methods incurs drastic losses in performance. To reduce the resolution further, in the extreme case even to binary weights, more advanced techniques are necessary. To this end, we propose two methods for mapping neural network algorithms with high resolution weights to corresponding algorithms that work with low resolution weights and demonstrate that their performance is substantially better than standard rounding. We further use these methods to investigate the performance of three common neural network algorithms under fixed memory size of the weight matrix with different weight resolutions. We show that dedicated hardware systems, whose technology dictates very low weight resolutions (be they electronic or biological) could in principle implement the algorithms we study.