Discrete Diffusion Language Model for Efficient Text Summarization

TL;DR

This paper introduces a novel discrete diffusion model with semantic-aware noising and CrossMamba architecture, enabling efficient and high-quality long-text summarization that outperforms existing models on key benchmarks.

Contribution

It presents a new semantic-aware noising process and an adapted CrossMamba model for improved long-text summarization with faster inference.

Findings

Achieved state-of-the-art ROUGE scores on benchmark datasets

Demonstrated faster inference speeds than autoregressive models

Successfully handled long sequences with Transformer backbones

Abstract

While diffusion models excel at conditional generating high-quality images, prior works in discrete diffusion models were not evaluated on conditional long-text generation. In this work, we address the limitations of prior discrete diffusion models for conditional long-text generation, particularly in long sequence-to-sequence tasks such as abstractive summarization. Despite fast decoding speeds compared to autoregressive methods, previous diffusion models failed on the abstractive summarization task due to the incompatibility between the backbone architectures and the random noising process. To overcome these challenges, we introduce a novel semantic-aware noising process that enables Transformer backbones to handle long sequences effectively. Additionally, we propose CrossMamba, an adaptation of the Mamba model to the encoder-decoder paradigm, which integrates seamlessly with the random absorbing noising process. Our approaches achieve state-of-the-art performance on three benchmark summarization datasets: Gigaword, CNN/DailyMail, and Arxiv, outperforming existing discrete diffusion models on ROUGE metrics as well as possessing much faster speed in inference compared to autoregressive models.