Reading and Reasoning over Chart Images for Evidence-based Automated Fact-Checking

TL;DR

This paper introduces a new task of chart-based fact-checking, proposing ChartBERT, a model that combines textual, structural, and visual chart information to verify claims, supported by a new dataset and extensive evaluation.

Contribution

The paper presents the first model for chart-based fact-checking, along with a new dataset and comprehensive evaluation of vision-language baselines.

Findings

ChartBERT outperforms existing vision-language models with 63.8% accuracy.

The task is complex but feasible, indicating promising directions for future research.

A new dataset, ChartFC, with 15,886 charts was created for this task.

Abstract

Evidence data for automated fact-checking (AFC) can be in multiple modalities such as text, tables, images, audio, or video. While there is increasing interest in using images for AFC, previous works mostly focus on detecting manipulated or fake images. We propose a novel task, chart-based fact-checking, and introduce ChartBERT as the first model for AFC against chart evidence. ChartBERT leverages textual, structural and visual information of charts to determine the veracity of textual claims. For evaluation, we create ChartFC, a new dataset of 15, 886 charts. We systematically evaluate 75 different vision-language (VL) baselines and show that ChartBERT outperforms VL models, achieving 63.8% accuracy. Our results suggest that the task is complex yet feasible, with many challenges ahead.