Multimodal Automated Fact-Checking: A Survey

TL;DR

This survey reviews the state of multimodal automated fact-checking, highlighting its unique challenges, existing benchmarks, models, and future research directions across text, image, audio, and video modalities.

Contribution

It introduces a comprehensive framework for multimodal automated fact-checking, encompassing subtasks and terminology, and surveys current benchmarks and models in this emerging field.

Findings

Multimodal misinformation spreads faster and is perceived as more credible.

Current models and benchmarks are limited and need further development.

Future research should address multimodal subtasks and improve model robustness.

Abstract

Misinformation is often conveyed in multiple modalities, e.g. a miscaptioned image. Multimodal misinformation is perceived as more credible by humans, and spreads faster than its text-only counterparts. While an increasing body of research investigates automated fact-checking (AFC), previous surveys mostly focus on text. In this survey, we conceptualise a framework for AFC including subtasks unique to multimodal misinformation. Furthermore, we discuss related terms used in different communities and map them to our framework. We focus on four modalities prevalent in real-world fact-checking: text, image, audio, and video. We survey benchmarks and models, and discuss limitations and promising directions for future research