News Summarization and Evaluation in the Era of GPT-3

TL;DR

This paper examines GPT-3's effectiveness in news summarization, showing it outperforms fine-tuned models in human preference and challenges existing evaluation metrics, while also exploring keyword-based summarization.

Contribution

It provides a comprehensive analysis of GPT-3's summarization capabilities, compares prompting to fine-tuning, and introduces new datasets and human judgments for evaluation.

Findings

GPT-3 summaries are preferred by humans over fine-tuned models.

Standard automatic metrics are unreliable for evaluating GPT-3 summaries.

Prompting with GPT-3 performs well in both generic and keyword-based summarization.

Abstract

The recent success of prompting large language models like GPT-3 has led to a paradigm shift in NLP research. In this paper, we study its impact on text summarization, focusing on the classic benchmark domain of news summarization. First, we investigate how GPT-3 compares against fine-tuned models trained on large summarization datasets. We show that not only do humans overwhelmingly prefer GPT-3 summaries, prompted using only a task description, but these also do not suffer from common dataset-specific issues such as poor factuality. Next, we study what this means for evaluation, particularly the role of gold standard test sets. Our experiments show that both reference-based and reference-free automatic metrics cannot reliably evaluate GPT-3 summaries. Finally, we evaluate models on a setting beyond generic summarization, specifically keyword-based summarization, and show how dominant fine-tuning approaches compare to prompting. To support further research, we release: (a) a corpus of 10K generated summaries from fine-tuned and prompt-based models across 4 standard summarization benchmarks, (b) 1K human preference judgments comparing different systems for generic- and keyword-based summarization.