Simplified TinyBERT: Knowledge Distillation for Document Retrieval

TL;DR

This paper introduces Simplified TinyBERT, a more efficient model for document retrieval that outperforms BERT-Base in speed and accuracy through knowledge distillation and model simplifications.

Contribution

The paper proposes two simplifications to TinyBERT and demonstrates their effectiveness in improving document ranking performance and efficiency.

Findings

Simplified TinyBERT outperforms BERT-Base in accuracy.

The model achieves 15× speedup over BERT-Base.

Knowledge distillation enhances document retrieval performance.

Abstract

Despite the effectiveness of utilizing the BERT model for document ranking, the high computational cost of such approaches limits their uses. To this end, this paper first empirically investigates the effectiveness of two knowledge distillation models on the document ranking task. In addition, on top of the recently proposed TinyBERT model, two simplifications are proposed. Evaluations on two different and widely-used benchmarks demonstrate that Simplified TinyBERT with the proposed simplifications not only boosts TinyBERT, but also significantly outperforms BERT-Base when providing 15$\times$ speedup.