Exploring Low-Cost Transformer Model Compression for Large-Scale Commercial Reply Suggestions

TL;DR

This paper investigates low-cost model compression methods like Layer Dropping and Layer Freezing to reduce training time for large-scale commercial reply suggestion systems, maintaining relevance and engagement.

Contribution

It demonstrates effective training time reduction using low-cost compression techniques in large-data scenarios without sacrificing model quality.

Findings

Training time reduced by 42%

Model relevance and user engagement unaffected

Robustness confirmed across different datasets and model sizes

Abstract

Fine-tuning pre-trained language models improves the quality of commercial reply suggestion systems, but at the cost of unsustainable training times. Popular training time reduction approaches are resource intensive, thus we explore low-cost model compression techniques like Layer Dropping and Layer Freezing. We demonstrate the efficacy of these techniques in large-data scenarios, enabling the training time reduction for a commercial email reply suggestion system by 42%, without affecting the model relevance or user engagement. We further study the robustness of these techniques to pre-trained model and dataset size ablation, and share several insights and recommendations for commercial applications.