Reading and Reasoning over Chart Images for Evidence-based Automated Fact-Checking
Mubashara Akhtar, Oana Cocarascu, Elena Simperl

TL;DR
This paper introduces a new task of chart-based fact-checking, proposing ChartBERT, a model that combines textual, structural, and visual chart information to verify claims, supported by a new dataset and extensive evaluation.
Contribution
The paper presents the first model for chart-based fact-checking, along with a new dataset and comprehensive evaluation of vision-language baselines.
Findings
ChartBERT outperforms existing vision-language models with 63.8% accuracy.
The task is complex but feasible, indicating promising directions for future research.
A new dataset, ChartFC, with 15,886 charts was created for this task.
Abstract
Evidence data for automated fact-checking (AFC) can be in multiple modalities such as text, tables, images, audio, or video. While there is increasing interest in using images for AFC, previous works mostly focus on detecting manipulated or fake images. We propose a novel task, chart-based fact-checking, and introduce ChartBERT as the first model for AFC against chart evidence. ChartBERT leverages textual, structural and visual information of charts to determine the veracity of textual claims. For evaluation, we create ChartFC, a new dataset of 15, 886 charts. We systematically evaluate 75 different vision-language (VL) baselines and show that ChartBERT outperforms VL models, achieving 63.8% accuracy. Our results suggest that the task is complex yet feasible, with many challenges ahead.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMultimodal Machine Learning Applications · Topic Modeling · Natural Language Processing Techniques
