Large Language Models and Provenance Metadata for Determining the Relevance of Images and Videos in News Stories

TL;DR

This paper presents a system built around a large language model that analyzes text and provenance metadata of images and videos to assess their relevance in news stories, aiming to combat multimodal misinformation.

Contribution

It introduces a novel multimodal approach combining language models and provenance metadata analysis for misinformation detection in news content.

Findings

Open-sourced prototype and web interface available

Effective in identifying relevant media in news stories

Addresses multimodal misinformation challenges

Abstract

The most effective misinformation campaigns are multimodal, often combining text with images and videos taken out of context -- or fabricating them entirely -- to support a given narrative. Contemporary methods for detecting misinformation, whether in deepfakes or text articles, often miss the interplay between multiple modalities. Built around a large language model, the system proposed in this paper addresses these challenges. It analyzes both the article's text and the provenance metadata of included images and videos to determine whether they are relevant. We open-source the system prototype and interactive web interface.