A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Models

TL;DR

This survey comprehensively reviews parallel text generation techniques, categorizing methods into autoregressive and non-autoregressive paradigms, analyzing their trade-offs, and outlining future research directions to improve inference efficiency in large language models.

Contribution

It provides a systematic taxonomy and analysis of parallel text generation methods, highlighting recent advancements and open challenges in the field.

Findings

Categorized approaches into AR-based and Non-AR-based methods.

Analyzed trade-offs between speed, quality, and efficiency.

Identified promising directions for future research.

Abstract

As text generation has become a core capability of modern Large Language Models (LLMs), it underpins a wide range of downstream applications. However, most existing LLMs rely on autoregressive (AR) generation, producing one token at a time based on previously generated context-resulting in limited generation speed due to the inherently sequential nature of the process. To address this challenge, an increasing number of researchers have begun exploring parallel text generation-a broad class of techniques aimed at breaking the token-by-token generation bottleneck and improving inference efficiency. Despite growing interest, there remains a lack of comprehensive analysis on what specific techniques constitute parallel text generation and how they improve inference performance. To bridge this gap, we present a systematic survey of parallel text generation methods. We categorize existing approaches into AR-based and Non-AR-based paradigms, and provide a detailed examination of the core techniques within each category. Following this taxonomy, we assess their theoretical trade-offs in terms of speed, quality, and efficiency, and examine their potential for combination and comparison with alternative acceleration strategies. Finally, based on our findings, we highlight recent advancements, identify open challenges, and outline promising directions for future research in parallel text generation. We have also created a GitHub repository for indexing relevant papers and open resources available at https://github.com/zhanglingzhe0820/Awesome-Parallel-Text-Generation.