Shortcomings of Question Answering Based Factuality Frameworks for Error Localization

TL;DR

This paper critically evaluates QA-based factuality metrics for error localization in summarization, revealing significant limitations and demonstrating that these frameworks often fail to accurately identify factual errors, outperforming trivial baselines.

Contribution

The study provides the first systematic analysis of QA-based factuality frameworks for error localization, highlighting their fundamental shortcomings and the propagation of errors from question generation modules.

Findings

QA-based frameworks fail to accurately localize error spans

Trivial exact match baselines outperform QA-based methods

Error propagation from question generation modules hampers localization

Abstract

Despite recent progress in abstractive summarization, models often generate summaries with factual errors. Numerous approaches to detect these errors have been proposed, the most popular of which are question answering (QA)-based factuality metrics. These have been shown to work well at predicting summary-level factuality and have potential to localize errors within summaries, but this latter capability has not been systematically evaluated in past research. In this paper, we conduct the first such analysis and find that, contrary to our expectations, QA-based frameworks fail to correctly identify error spans in generated summaries and are outperformed by trivial exact match baselines. Our analysis reveals a major reason for such poor localization: questions generated by the QG module often inherit errors from non-factual summaries which are then propagated further into downstream modules. Moreover, even human-in-the-loop question generation cannot easily offset these problems. Our experiments conclusively show that there exist fundamental issues with localization using the QA framework which cannot be fixed solely by stronger QA and QG models.