Verifying Text Summaries of Relational Data Sets

TL;DR

The paper introduces the AggChecker, a probabilistic tool for verifying the accuracy of natural language claims in text summaries of relational data, improving fact-checking efficiency and accuracy.

Contribution

It presents a novel probabilistic model and system for automatically verifying text claims against data, enabling faster and more accurate fact-checking of summaries.

Findings

Revealed errors in about one-third of tested articles.

Users checked summaries six times faster with AggChecker.

Achieved higher recall and precision than existing baselines.

Abstract

We present a novel natural language query interface, the AggChecker, aimed at text summaries of relational data sets. The tool focuses on natural language claims that translate into an SQL query and a claimed query result. Similar in spirit to a spell checker, the AggChecker marks up text passages that seem to be inconsistent with the actual data. At the heart of the system is a probabilistic model that reasons about the input document in a holistic fashion. Based on claim keywords and the document structure, it maps each text claim to a probability distribution over associated query translations. By efficiently executing tens to hundreds of thousands of candidate translations for a typical input document, the system maps text claims to correctness probabilities. This process becomes practical via a specialized processing backend, avoiding redundant work via query merging and result caching. Verification is an interactive process in which users are shown tentative results, enabling them to take corrective actions if necessary. Our system was tested on a set of 53 public articles containing 392 claims. Our test cases include articles from major newspapers, summaries of survey results, and Wikipedia articles. Our tool revealed erroneous claims in roughly a third of test cases. A detailed user study shows that users using our tool are in average six times faster at checking text summaries, compared to generic SQL interfaces. In fully automated verification, our tool achieves significantly higher recall and precision than baselines from the areas of natural language query interfaces and fact-checking.