Model Compression with Two-stage Multi-teacher Knowledge Distillation for Web Question Answering System

TL;DR

This paper introduces a two-stage multi-teacher knowledge distillation approach to compress large question answering models, significantly improving inference speed while maintaining high accuracy.

Contribution

The proposed TMKD method effectively reduces model size and inference time without sacrificing performance by combining pre-training and multi-teacher fine-tuning.

Findings

Significant speedup in model inference.

Outperforms baseline compression methods.

Achieves comparable accuracy to original models.

Abstract

Deep pre-training and fine-tuning models (such as BERT and OpenAI GPT) have demonstrated excellent results in question answering areas. However, due to the sheer amount of model parameters, the inference speed of these models is very slow. How to apply these complex models to real business scenarios becomes a challenging but practical problem. Previous model compression methods usually suffer from information loss during the model compression procedure, leading to inferior models compared with the original one. To tackle this challenge, we propose a Two-stage Multi-teacher Knowledge Distillation (TMKD for short) method for web Question Answering system. We first develop a general Q\&A distillation task for student model pre-training, and further fine-tune this pre-trained student model with multi-teacher knowledge distillation on downstream tasks (like Web Q\&A task, MNLI, SNLI, RTE tasks from GLUE), which effectively reduces the overfitting bias in individual teacher models, and transfers more general knowledge to the student model. The experiment results show that our method can significantly outperform the baseline methods and even achieve comparable results with the original teacher models, along with substantial speedup of model inference.