Pyramid-BERT: Reducing Complexity via Successive Core-set based Token Selection

TL;DR

Pyramid-BERT introduces a core-set based token selection method to reduce the computational complexity of BERT, enabling efficient processing of longer sequences without pre-training, and demonstrates superior performance on NLP benchmarks.

Contribution

The paper proposes a novel core-set based token selection technique for BERT, replacing heuristics, to improve efficiency and handle longer sequences effectively.

Findings

Outperforms baselines on GLUE benchmarks

Achieves better results on Long Range Arena datasets

Enables longer sequence processing without pre-training

Abstract

Transformer-based language models such as BERT have achieved the state-of-the-art performance on various NLP tasks, but are computationally prohibitive. A recent line of works use various heuristics to successively shorten sequence length while transforming tokens through encoders, in tasks such as classification and ranking that require a single token embedding for prediction. We present a novel solution to this problem, called Pyramid-BERT where we replace previously used heuristics with a {\em core-set} based token selection method justified by theoretical results. The core-set based token selection technique allows us to avoid expensive pre-training, gives a space-efficient fine tuning, and thus makes it suitable to handle longer sequence lengths. We provide extensive experiments establishing advantages of pyramid BERT over several baselines and existing works on the GLUE benchmarks and Long Range Arena datasets.