Selecting Data to Clean for Fact Checking: Minimizing Uncertainty vs. Maximizing Surprise

TL;DR

This paper investigates how to optimally select data to clean for fact-checking, balancing between reducing uncertainty and maximizing the chance to counter false claims, with new algorithms for complex, non-linear objectives.

Contribution

It introduces a formal framework for data cleaning in fact-checking, analyzing different objectives and providing efficient algorithms for complex optimization problems.

Findings

Objectives of minimizing uncertainty and maximizing counterability can conflict.

Efficient algorithms outperform naive solutions for complex data cleaning tasks.

Results generalize to a broad class of functions with applications beyond fact-checking.

Abstract

We study the optimization problem of selecting numerical quantities to clean in order to fact-check claims based on such data. Oftentimes, such claims are technically correct, but they can still mislead for two reasons. First, data may contain uncertainty and errors. Second, data can be "fished" to advance particular positions. In practice, fact-checkers cannot afford to clean all data and must choose to clean what "matters the most" to checking a claim. We explore alternative definitions of what "matters the most": one is to ascertain claim qualities (by minimizing uncertainty in these measures), while an alternative is just to counter the claim (by maximizing the probability of finding a counterargument). We show whether the two objectives align with each other, with important implications on when fact-checkers should exercise care in selective data cleaning, to avoid potential bias introduced by their desire to counter claims. We develop efficient algorithms for solving the various variants of the optimization problem, showing significant improvements over naive solutions. The problem is particularly challenging because the objectives in the fact-checking context are complex, non-linear functions over data. We obtain results that generalize to a large class of functions, with potential applications beyond fact-checking.