Generating Wikipedia by Summarizing Long Sequences

TL;DR

This paper presents a scalable decoder-only neural model for generating Wikipedia articles by summarizing long sequences of source documents, combining extractive and abstractive methods to produce coherent, factual content.

Contribution

It introduces a novel decoder-only architecture capable of attending to very long sequences for multi-document summarization and Wikipedia article generation.

Findings

The model generates fluent, coherent multi-sentence paragraphs.

It effectively extracts relevant factual information from source documents.

Performance is validated through perplexity, ROUGE scores, and human evaluations.

Abstract

We show that generating English Wikipedia articles can be approached as a multi- document summarization of source documents. We use extractive summarization to coarsely identify salient information and a neural abstractive model to generate the article. For the abstractive model, we introduce a decoder-only architecture that can scalably attend to very long sequences, much longer than typical encoder- decoder architectures used in sequence transduction. We show that this model can generate fluent, coherent multi-sentence paragraphs and even whole Wikipedia articles. When given reference documents, we show it can extract relevant factual information as reflected in perplexity, ROUGE scores and human evaluations.