Can depth-adaptive BERT perform better on binary classification tasks

TL;DR

This paper investigates whether smaller, depth-adaptive BERT models can outperform the full model on binary classification tasks and proposes a simple method to efficiently shrink BERT without significant accuracy loss.

Contribution

It introduces a straightforward approach to identify and create smaller, more efficient BERT sub-networks that maintain or improve performance on binary classification tasks.

Findings

Smaller BERT sub-networks can outperform the full model.

The proposed shrinking method reduces time and storage with minimal accuracy loss.

Effective model compression without complex calculations.

Abstract

In light of the success of transferring language models into NLP tasks, we ask whether the full BERT model is always the best and does it exist a simple but effective method to find the winning ticket in state-of-the-art deep neural networks without complex calculations. We construct a series of BERT-based models with different size and compare their predictions on 8 binary classification tasks. The results show there truly exist smaller sub-networks performing better than the full model. Then we present a further study and propose a simple method to shrink BERT appropriately before fine-tuning. Some extended experiments indicate that our method could save time and storage overhead extraordinarily with little even no accuracy loss.