Verifying Cross-modal Entity Consistency in News using Vision-language Models

TL;DR

This paper introduces LVLM4CEC, a framework using large vision-language models to verify entity consistency across images and text in news, helping to detect disinformation by assessing persons, locations, and events.

Contribution

It proposes novel prompting strategies for LVLMs in news entity verification and extends datasets with manual ground-truth data for this task.

Findings

Improved accuracy in verifying persons and events with evidence images.

Outperforms baseline methods in location and event verification.

Demonstrates potential of LVLMs for automating cross-modal entity verification.

Abstract

The web has become a crucial source of information, but it is also used to spread disinformation, often conveyed through multiple modalities like images and text. The identification of inconsistent cross-modal information, in particular entities such as persons, locations, and events, is critical to detect disinformation. Previous works either identify out-of-context disinformation by assessing the consistency of images to the whole document, neglecting relations of individual entities, or focus on generic entities that are not relevant to news. So far, only few approaches have addressed the task of validating entity consistency between images and text in news. However, the potential of large vision-language models (LVLMs) has not been explored yet. In this paper, we propose an LVLM-based framework for verifying Cross-modal Entity Consistency~(LVLM4CEC), to assess whether persons, locations and events in news articles are consistent across both modalities. We suggest effective prompting strategies for LVLMs for entity verification that leverage reference images crawled from web. Moreover, we extend three existing datasets for the task of entity verification in news providing manual ground-truth data. Our results show the potential of LVLMs for automating cross-modal entity verification, showing improved accuracy in identifying persons and events when using evidence images. Moreover, our method outperforms a baseline for location and event verification in documents. The datasets and source code are available on GitHub at https://github.com/TIBHannover/LVLM4CEC.