FastSeq: Make Sequence Generation Faster

TL;DR

FastSeq is a framework that significantly accelerates sequence generation in Transformer models by employing various optimization techniques, achieving 4-9x faster inference without sacrificing accuracy.

Contribution

The paper introduces FastSeq, a set of optimization methods that improve inference speed of Transformer-based models with minimal implementation effort.

Findings

Achieves 4-9x inference speedup on multiple models

Maintains accuracy while accelerating generation

Easy to integrate with a simple code change

Abstract

Transformer-based models have made tremendous impacts in natural language generation. However the inference speed is a bottleneck due to large model size and intensive computing involved in auto-regressive decoding process. We develop FastSeq framework to accelerate sequence generation without accuracy loss. The proposed optimization techniques include an attention cache optimization, an efficient algorithm for detecting repeated n-grams, and an asynchronous generation pipeline with parallel I/O. These optimizations are general enough to be applicable to Transformer-based models (e.g., T5, GPT2, and UniLM). Our benchmark results on a set of widely used and diverse models demonstrate 4-9x inference speed gain. Additionally, FastSeq is easy to use with a simple one-line code change. The source code is available at https://github.com/microsoft/fastseq.