Fast DistilBERT on CPUs

TL;DR

This paper introduces a pipeline for creating fast, efficient Transformer models like DistilBERT optimized for CPU deployment, achieving significant speedups with minimal accuracy loss for NLP tasks.

Contribution

The authors present a novel CPU-optimized pipeline combining hardware-aware pruning, knowledge distillation, quantization, and a custom runtime engine for Transformer inference.

Findings

Up to 4.1x speedup over ONNX Runtime.

Performance exceeds Neural Magic's DeepSparse by 50%.

Minimal accuracy loss on SQuADv1.1 benchmark.

Abstract

Transformer-based language models have become the standard approach to solving natural language processing tasks. However, industry adoption usually requires the maximum throughput to comply with certain latency constraints that prevents Transformer models from being used in production. To address this gap, model compression techniques such as quantization and pruning may be used to improve inference efficiency. However, these compression techniques require specialized software to apply and deploy at scale. In this work, we propose a new pipeline for creating and running Fast Transformer models on CPUs, utilizing hardware-aware pruning, knowledge distillation, quantization, and our own Transformer inference runtime engine with optimized kernels for sparse and quantized operators. We demonstrate the efficiency of our pipeline by creating a Fast DistilBERT model showing minimal accuracy loss on the question-answering SQuADv1.1 benchmark, and throughput results under typical production constraints and environments. Our results outperform existing state-of-the-art Neural Magic's DeepSparse runtime performance by up to 50% and up to 4.1x performance speedup over ONNX Runtime. Source code is publicly available at https://github.com/intel/intel-extension-for-transformers.