Investigating Crowdsourcing Protocols for Evaluating the Factual Consistency of Summaries

TL;DR

This paper compares different crowdsourcing evaluation protocols to determine the most reliable method for assessing the factual consistency of summaries generated by AI models, highlighting the superiority of ranking-based methods.

Contribution

It systematically evaluates and compares Likert scale and ranking-based crowdsourcing protocols for factual consistency assessment in summarization.

Findings

Ranking-based protocols are more reliable across datasets.

Likert ratings' reliability varies with dataset and design.

Crowdsourcing templates and evaluations will be publicly available.

Abstract

Current pre-trained models applied to summarization are prone to factual inconsistencies which either misrepresent the source text or introduce extraneous information. Thus, comparing the factual consistency of summaries is necessary as we develop improved models. However, the optimal human evaluation setup for factual consistency has not been standardized. To address this issue, we crowdsourced evaluations for factual consistency using the rating-based Likert scale and ranking-based Best-Worst Scaling protocols, on 100 articles from each of the CNN-Daily Mail and XSum datasets over four state-of-the-art models, to determine the most reliable evaluation framework. We find that ranking-based protocols offer a more reliable measure of summary quality across datasets, while the reliability of Likert ratings depends on the target dataset and the evaluation design. Our crowdsourcing templates and summary evaluations will be publicly available to facilitate future research on factual consistency in summarization.