PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization

TL;DR

PRIMERA is a novel pre-trained model designed for multi-document summarization that effectively captures and aggregates information across documents, reducing the need for extensive fine-tuning and dataset-specific adjustments.

Contribution

It introduces a new pre-training objective and an efficient transformer architecture tailored for multi-document summarization tasks.

Findings

Outperforms state-of-the-art models on multiple datasets

Effective in zero-shot, few-shot, and full-supervised settings

Reduces reliance on large labeled datasets

Abstract

We introduce PRIMERA, a pre-trained model for multi-document representation with a focus on summarization that reduces the need for dataset-specific architectures and large amounts of fine-tuning labeled data. PRIMERA uses our newly proposed pre-training objective designed to teach the model to connect and aggregate information across documents. It also uses efficient encoder-decoder transformers to simplify the processing of concatenated input documents. With extensive experiments on 6 multi-document summarization datasets from 3 different domains on zero-shot, few-shot and full-supervised settings, PRIMERA outperforms current state-of-the-art dataset-specific and pre-trained models on most of these settings with large margins. The code and pre-trained models can be found at \url{https://github.com/allenai/PRIMER}.