Pipelined Decoder for Efficient Context-Aware Text Generation

TL;DR

This paper introduces a pipelined decoder architecture that enables parallel, context-aware text generation, significantly improving speed while maintaining quality across various NLP tasks.

Contribution

A novel pipelined decoder design that allows simultaneous generation of multiple subsequences, enhancing speed without sacrificing quality or increasing memory use.

Findings

Significant speedup in text generation tasks.

Maintains comparable quality to traditional autoregressive models.

Effective across multiple NLP applications.

Abstract

As the basis of generative AI, an autoregressive model requires the generation of a new token depending on all the previously generated tokens, which brings high quality but also restricts the model to generate tokens one by one, forming a bottleneck limiting the generation speed. In this paper, we propose a new decoder architecture that efficiently generates text in parallel for context-aware generation tasks. Our proposed pipelined decoder initiates the generation of multiple subsequences simultaneously, and, at each time-step, it generates a new token for each subsequence to realize parallelism. Experiments on multiple text generation tasks, including question answering, text summarization, and keyphrase generation, show that our pipelined decoder significantly improves the generation speed without a significant loss of generation quality or additional memory consumption.