"The Data Says Otherwise"-Towards Automated Fact-checking and Communication of Data Claims

TL;DR

This paper introduces Aletheia, an automated fact-checking tool that uses large language models to verify data claims and communicate evidence through tables and visualizations, aiming to improve efficiency and user understanding.

Contribution

The work presents a novel integrated system combining NLP, data visualization, and interaction design for automated data claim verification and evidence communication.

Findings

Visualizations improve user confidence and assessment speed.

LLMs are effective but have limitations in data fact-checking.

Design recommendations enhance data evidence presentation.

Abstract

Fact-checking data claims requires data evidence retrieval and analysis, which can become tedious and intractable when done manually. This work presents Aletheia, an automated fact-checking prototype designed to facilitate data claims verification and enhance data evidence communication. For verification, we utilize a pre-trained LLM to parse the semantics for evidence retrieval. To effectively communicate the data evidence, we design representations in two forms: data tables and visualizations, tailored to various data fact types. Additionally, we design interactions that showcase a real-world application of these techniques. We evaluate the performance of two core NLP tasks with a curated dataset comprising 400 data claims and compare the two representation forms regarding viewers' assessment time, confidence, and preference via a user study with 20 participants. The evaluation offers insights into the feasibility and bottlenecks of using LLMs for data fact-checking tasks, potential advantages and disadvantages of using visualizations over data tables, and design recommendations for presenting data evidence.