Latent Weights Do Not Exist: Rethinking Binarized Neural Network Optimization

TL;DR

This paper challenges the traditional view of latent weights in Binarized Neural Networks, proposing they serve as inertia rather than real weights, and introduces a new optimizer, Bop, tailored for BNN training.

Contribution

It redefines the role of latent weights in BNNs as inertia and introduces Bop, the first optimizer specifically designed for BNNs, improving training understanding and performance.

Findings

Bop outperforms existing optimizers on CIFAR-10 and ImageNet.

Redefining latent weights as inertia enhances BNN optimization understanding.

The new perspective facilitates future improvements in BNN training methods.

Abstract

Optimization of Binarized Neural Networks (BNNs) currently relies on real-valued latent weights to accumulate small update steps. In this paper, we argue that these latent weights cannot be treated analogously to weights in real-valued networks. Instead their main role is to provide inertia during training. We interpret current methods in terms of inertia and provide novel insights into the optimization of BNNs. We subsequently introduce the first optimizer specifically designed for BNNs, Binary Optimizer (Bop), and demonstrate its performance on CIFAR-10 and ImageNet. Together, the redefinition of latent weights as inertia and the introduction of Bop enable a better understanding of BNN optimization and open up the way for further improvements in training methodologies for BNNs. Code is available at: https://github.com/plumerai/rethinking-bnn-optimization