Multi-CLS BERT: An Efficient Alternative to Traditional Ensembling

TL;DR

Multi-CLS BERT introduces a more efficient ensembling method using multiple CLS tokens, achieving comparable accuracy to traditional ensembles with significantly reduced computation and memory requirements.

Contribution

It proposes a novel Multi-CLS BERT approach that enables effective ensembling within a single model, reducing resource costs while maintaining high performance.

Findings

Improves accuracy and confidence estimation on GLUE and SuperGLUE

Outperforms BERT_Large with only 100 training samples in GLUE

Reduces computation and memory by nearly 4 times compared to traditional ensembles

Abstract

Ensembling BERT models often significantly improves accuracy, but at the cost of significantly more computation and memory footprint. In this work, we propose Multi-CLS BERT, a novel ensembling method for CLS-based prediction tasks that is almost as efficient as a single BERT model. Multi-CLS BERT uses multiple CLS tokens with a parameterization and objective that encourages their diversity. Thus instead of fine-tuning each BERT model in an ensemble (and running them all at test time), we need only fine-tune our single Multi-CLS BERT model (and run the one model at test time, ensembling just the multiple final CLS embeddings). To test its effectiveness, we build Multi-CLS BERT on top of a state-of-the-art pretraining method for BERT (Aroca-Ouellette and Rudzicz, 2020). In experiments on GLUE and SuperGLUE we show that our Multi-CLS BERT reliably improves both overall accuracy and confidence estimation. When only 100 training samples are available in GLUE, the Multi-CLS BERT_Base model can even outperform the corresponding BERT_Large model. We analyze the behavior of our Multi-CLS BERT, showing that it has many of the same characteristics and behavior as a typical BERT 5-way ensemble, but with nearly 4-times less computation and memory.