Random Bias Initialization Improves Quantized Training

TL;DR

This paper investigates the performance gap in binary neural networks compared to full-precision models, revealing that random bias initialization can significantly improve binary training accuracy by better understanding network geometry.

Contribution

It introduces the insight that random bias initialization enhances binary neural network training, addressing the accuracy gap with a novel perspective on network geometry.

Findings

Random bias initialization improves binary network accuracy.

Comparison reveals differences in network geometry between full-precision and binary models.

Proposes a simple yet effective initialization strategy for binary neural networks.

Abstract

Binary neural networks improve computationally efficiency of deep models with a large margin. However, there is still a performance gap between a successful full-precision training and binary training. We bring some insights about why this accuracy drop exists and call for a better understanding of binary network geometry. We start with analyzing full-precision neural networks with ReLU activation and compare it with its binarized version. This comparison suggests to initialize networks with random bias, a counter-intuitive remedy.