BiBERT: Accurate Fully Binarized BERT

TL;DR

BiBERT is a fully binarized BERT model that significantly reduces computation and memory costs while maintaining high performance, achieved through novel attention and distillation techniques.

Contribution

This paper introduces BiBERT, the first fully binarized BERT, with new attention and distillation methods to overcome performance drops in binarization.

Findings

Outperforms existing quantized BERTs on NLP benchmarks

Achieves 56.3x FLOPs and 31.2x model size reduction

Maintains competitive accuracy with ultra-low bit activations

Abstract

The large pre-trained BERT has achieved remarkable performance on Natural Language Processing (NLP) tasks but is also computation and memory expensive. As one of the powerful compression approaches, binarization extremely reduces the computation and memory consumption by utilizing 1-bit parameters and bitwise operations. Unfortunately, the full binarization of BERT (i.e., 1-bit weight, embedding, and activation) usually suffer a significant performance drop, and there is rare study addressing this problem. In this paper, with the theoretical justification and empirical analysis, we identify that the severe performance drop can be mainly attributed to the information degradation and optimization direction mismatch respectively in the forward and backward propagation, and propose BiBERT, an accurate fully binarized BERT, to eliminate the performance bottlenecks. Specifically, BiBERT introduces an efficient Bi-Attention structure for maximizing representation information statistically and a Direction-Matching Distillation (DMD) scheme to optimize the full binarized BERT accurately. Extensive experiments show that BiBERT outperforms both the straightforward baseline and existing state-of-the-art quantized BERTs with ultra-low bit activations by convincing margins on the NLP benchmark. As the first fully binarized BERT, our method yields impressive 56.3 times and 31.2 times saving on FLOPs and model size, demonstrating the vast advantages and potential of the fully binarized BERT model in real-world resource-constrained scenarios.