Lipschitz Continuity Retained Binary Neural Network

TL;DR

This paper introduces a Lipschitz continuity-based regularization for binary neural networks (BNNs) to enhance robustness and performance, addressing the challenge of quantization error and spectral norm approximation.

Contribution

It proposes a novel Lipschitz regularization method with Retention Matrices for BNNs, improving robustness without exact spectral norm computation.

Findings

Achieves state-of-the-art results on CIFAR and ImageNet.

Effectively improves BNN robustness on ImageNet-C.

Outperforms existing binarization techniques.

Abstract

Relying on the premise that the performance of a binary neural network can be largely restored with eliminated quantization error between full-precision weight vectors and their corresponding binary vectors, existing works of network binarization frequently adopt the idea of model robustness to reach the aforementioned objective. However, robustness remains to be an ill-defined concept without solid theoretical support. In this work, we introduce the Lipschitz continuity, a well-defined functional property, as the rigorous criteria to define the model robustness for BNN. We then propose to retain the Lipschitz continuity as a regularization term to improve the model robustness. Particularly, while the popular Lipschitz-involved regularization methods often collapse in BNN due to its extreme sparsity, we design the Retention Matrices to approximate spectral norms of the targeted weight matrices, which can be deployed as the approximation for the Lipschitz constant of BNNs without the exact Lipschitz constant computation (NP-hard). Our experiments prove that our BNN-specific regularization method can effectively strengthen the robustness of BNN (testified on ImageNet-C), achieving state-of-the-art performance on CIFAR and ImageNet.