Deploying a BERT-based Query-Title Relevance Classifier in a Production System: a View from the Trenches

TL;DR

This paper presents a compact BERT-based model, BertBiLSTM, optimized for low-latency production environments, outperforming standard BERT in accuracy and inference speed for query-title relevance tasks.

Contribution

The authors develop a novel, efficient model called BertBiLSTM, trained via knowledge distillation and data augmentation, suitable for real-time industrial NLP applications.

Findings

BertBiLSTM infers in at most 0.2ms on CPU.

BertBiLSTM surpasses off-the-shelf BERT in accuracy.

The model outperforms other compact models in production settings.

Abstract

The Bidirectional Encoder Representations from Transformers (BERT) model has been radically improving the performance of many Natural Language Processing (NLP) tasks such as Text Classification and Named Entity Recognition (NER) applications. However, it is challenging to scale BERT for low-latency and high-throughput industrial use cases due to its enormous size. We successfully optimize a Query-Title Relevance (QTR) classifier for deployment via a compact model, which we name BERT Bidirectional Long Short-Term Memory (BertBiLSTM). The model is capable of inferring an input in at most 0.2ms on CPU. BertBiLSTM exceeds the off-the-shelf BERT model's performance in terms of accuracy and efficiency for the aforementioned real-world production task. We achieve this result in two phases. First, we create a pre-trained model, called eBERT, which is the original BERT architecture trained with our unique item title corpus. We then fine-tune eBERT for the QTR task. Second, we train the BertBiLSTM model to mimic the eBERT model's performance through a process called Knowledge Distillation (KD) and show the effect of data augmentation to achieve the resembling goal. Experimental results show that the proposed model outperforms other compact and production-ready models.