Optimizing Inference Performance of Transformers on CPUs

TL;DR

This paper analyzes the inference performance of Transformer models on CPUs, identifies bottlenecks, and proposes three optimizations that double the speed without affecting accuracy.

Contribution

It provides an empirical analysis of Transformer inference on CPUs and introduces three optimizations to significantly improve speed without model modifications.

Findings

Achieved up to 2.37x speedup in inference

Identified key computation bottlenecks in Transformer architecture

Optimizations do not alter model accuracy

Abstract

The Transformer architecture revolutionized the field of natural language processing (NLP). Transformers-based models (e.g., BERT) power many important Web services, such as search, translation, question-answering, etc. While enormous research attention is paid to the training of those models, relatively little efforts are made to improve their inference performance. This paper comes to address this gap by presenting an empirical analysis of scalability and performance of inferencing a Transformer-based model on CPUs. Focusing on the highly popular BERT model, we identify key components of the Transformer architecture where the bulk of the computation happens, and propose three optimizations to speed them up. The optimizations are evaluated using the inference benchmark from HuggingFace, and are shown to achieve the speedup of up to x2.37. The considered optimizations do not require any changes to the implementation of the models nor affect their accuracy.