Bag of Tricks for Optimizing Transformer Efficiency

TL;DR

This paper demonstrates that simple, hardware-agnostic techniques like hyper-parameter tuning and design choices can significantly enhance Transformer efficiency, achieving up to 3.80X speedup on CPU and 2.52X on GPU.

Contribution

It introduces a set of straightforward, hardware-independent methods to optimize Transformer models, improving efficiency without complex modifications.

Findings

3.80X inference speedup on CPU

2.52X inference speedup on GPU

Simple techniques outperform complex methods in efficiency gains

Abstract

Improving Transformer efficiency has become increasingly attractive recently. A wide range of methods has been proposed, e.g., pruning, quantization, new architectures and etc. But these methods are either sophisticated in implementation or dependent on hardware. In this paper, we show that the efficiency of Transformer can be improved by combining some simple and hardware-agnostic methods, including tuning hyper-parameters, better design choices and training strategies. On the WMT news translation tasks, we improve the inference efficiency of a strong Transformer system by 3.80X on CPU and 2.52X on GPU. The code is publicly available at https://github.com/Lollipop321/mini-decoder-network.