Evaluating the Tradeoff Between Abstractiveness and Factuality in Abstractive Summarization

TL;DR

This paper investigates how increasing abstractiveness in neural abstractive summarization affects factual accuracy, analyzing multiple datasets and models through human evaluations and proposing new metrics to measure factuality adjusted for abstractiveness.

Contribution

It introduces datasets with human factuality judgments, visualizes the tradeoff between abstractiveness and factuality, and proposes new metrics to evaluate factuality considering abstractiveness.

Findings

Higher abstractiveness generally reduces factuality.

Factuality decay rate varies with training data.

New metrics effectively adjust factuality scores for abstractiveness.

Abstract

Neural models for abstractive summarization tend to generate output that is fluent and well-formed but lacks semantic faithfulness, or factuality, with respect to the input documents. In this paper, we analyze the tradeoff between abstractiveness and factuality of generated summaries across multiple datasets and models, using extensive human evaluations of factuality. In our analysis, we visualize the rates of change in factuality as we gradually increase abstractiveness using a decoding constraint, and we observe that, while increased abstractiveness generally leads to a drop in factuality, the rate of factuality decay depends on factors such as the data that the system was trained on. We introduce two datasets with human factuality judgements; one containing 10.2k generated summaries with systematically varied degrees of abstractiveness; the other containing 4.2k summaries from five different summarization models. We propose new factuality metrics that adjust for the degree of abstractiveness, and we use them to compare the abstractiveness-adjusted factuality of previous summarization works, providing baselines for future work.