Deep Multimodal Image-Repurposing Detection

TL;DR

This paper introduces a new challenging dataset and a deep multimodal model for detecting image repurposing by analyzing images and related metadata, significantly improving detection accuracy over existing methods.

Contribution

The paper presents the MEIR dataset for image repurposing detection and a novel end-to-end deep multimodal model that outperforms state-of-the-art techniques.

Findings

The proposed model achieves up to 0.23 higher AUC on MEIR.

It outperforms existing methods on multiple datasets.

The dataset includes real-world manipulations of location, person, and organization.

Abstract

Nefarious actors on social media and other platforms often spread rumors and falsehoods through images whose metadata (e.g., captions) have been modified to provide visual substantiation of the rumor/falsehood. This type of modification is referred to as image repurposing, in which often an unmanipulated image is published along with incorrect or manipulated metadata to serve the actor's ulterior motives. We present the Multimodal Entity Image Repurposing (MEIR) dataset, a substantially challenging dataset over that which has been previously available to support research into image repurposing detection. The new dataset includes location, person, and organization manipulations on real-world data sourced from Flickr. We also present a novel, end-to-end, deep multimodal learning model for assessing the integrity of an image by combining information extracted from the image with related information from a knowledge base. The proposed method is compared against state-of-the-art techniques on existing datasets as well as MEIR, where it outperforms existing methods across the board, with AUC improvement up to 0.23.