BEBERT: Efficient and Robust Binary Ensemble BERT

TL;DR

BEBERT introduces an ensemble of binary BERT models that significantly improves accuracy and robustness while maintaining computational efficiency, reducing training time and model size compared to full-precision BERT.

Contribution

This work is the first to apply ensemble techniques to binary BERT models, achieving superior accuracy and robustness without knowledge distillation, and demonstrating practical efficiency gains.

Findings

BEBERT outperforms existing binary BERT models in accuracy and robustness.

BEBERT achieves a 2x speedup in training time.

BEBERT reduces model size by 13x and FLOPs by 15x with minimal accuracy loss.

Abstract

Pre-trained BERT models have achieved impressive accuracy on natural language processing (NLP) tasks. However, their excessive amount of parameters hinders them from efficient deployment on edge devices. Binarization of the BERT models can significantly alleviate this issue but comes with a severe accuracy drop compared with their full-precision counterparts. In this paper, we propose an efficient and robust binary ensemble BERT (BEBERT) to bridge the accuracy gap. To the best of our knowledge, this is the first work employing ensemble techniques on binary BERTs, yielding BEBERT, which achieves superior accuracy while retaining computational efficiency. Furthermore, we remove the knowledge distillation procedures during ensemble to speed up the training process without compromising accuracy. Experimental results on the GLUE benchmark show that the proposed BEBERT significantly outperforms the existing binary BERT models in accuracy and robustness with a 2x speedup on training time. Moreover, our BEBERT has only a negligible accuracy loss of 0.3% compared to the full-precision baseline while saving 15x and 13x in FLOPs and model size, respectively. In addition, BEBERT also outperforms other compressed BERTs in accuracy by up to 6.7%.