Shortcomings of Question Answering Based Factuality Frameworks for Error Localization
Ryo Kamoi, Tanya Goyal, Greg Durrett

TL;DR
This paper critically evaluates QA-based factuality metrics for error localization in summarization, revealing significant limitations and demonstrating that these frameworks often fail to accurately identify factual errors, outperforming trivial baselines.
Contribution
The study provides the first systematic analysis of QA-based factuality frameworks for error localization, highlighting their fundamental shortcomings and the propagation of errors from question generation modules.
Findings
QA-based frameworks fail to accurately localize error spans
Trivial exact match baselines outperform QA-based methods
Error propagation from question generation modules hampers localization
Abstract
Despite recent progress in abstractive summarization, models often generate summaries with factual errors. Numerous approaches to detect these errors have been proposed, the most popular of which are question answering (QA)-based factuality metrics. These have been shown to work well at predicting summary-level factuality and have potential to localize errors within summaries, but this latter capability has not been systematically evaluated in past research. In this paper, we conduct the first such analysis and find that, contrary to our expectations, QA-based frameworks fail to correctly identify error spans in generated summaries and are outperformed by trivial exact match baselines. Our analysis reveals a major reason for such poor localization: questions generated by the QG module often inherit errors from non-factual summaries which are then propagated further into downstream…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Natural Language Processing Techniques · Advanced Graph Neural Networks
