Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectors

TL;DR

This paper evaluates the performance of various factuality metrics, including ChatGPT-based ones, across different summarization models and error types, revealing significant variability and the need for nuanced evaluation practices.

Contribution

It aggregates and stratifies factuality error annotations from multiple datasets, compares state-of-the-art metrics across models, and provides insights into their varying effectiveness and best practices.

Findings

Performance of factuality metrics varies across models and error types.

Recent improvements are mostly on older models, not recent ones.

No single metric outperforms others in all settings.

Abstract

The propensity of abstractive summarization models to make factual errors has been studied extensively, including design of metrics to detect factual errors and annotation of errors in current systems' outputs. However, the ever-evolving nature of summarization systems, metrics, and annotated benchmarks makes factuality evaluation a moving target, and drawing clear comparisons among metrics has become increasingly difficult. In this work, we aggregate factuality error annotations from nine existing datasets and stratify them according to the underlying summarization model. We compare performance of state-of-the-art factuality metrics, including recent ChatGPT-based metrics, on this stratified benchmark and show that their performance varies significantly across different types of summarization models. Critically, our analysis shows that much of the recent improvement in the factuality detection space has been on summaries from older (pre-Transformer) models instead of more relevant recent summarization models. We further perform a finer-grained analysis per error-type and find similar performance variance across error types for different factuality metrics. Our results show that no one metric is superior in all settings or for all error types, and we provide recommendations for best practices given these insights.