Deep Learning Methods for Event Verification and Image Repurposing Detection

TL;DR

This paper explores deep learning techniques to verify the authenticity of images by identifying the events they depict, using pre-trained networks and various classification strategies, demonstrating effectiveness in event verification and image re-purposing detection.

Contribution

The paper introduces novel deep learning methods for event verification using pre-trained networks, including feature extraction and fine-tuning strategies, for the first time applied to this problem.

Findings

Global scale classification marginally outperforms local methods.

Fine-tuning pre-trained networks improves accuracy.

Both proposed approaches are effective for event verification.

Abstract

The authenticity of images posted on social media is an issue of growing concern. Many algorithms have been developed to detect manipulated images, but few have investigated the ability of deep neural network based approaches to verify the authenticity of image labels, such as event names. In this paper, we propose several novel methods to predict if an image was captured at one of several noteworthy events. We use a set of images from several recorded events such as storms, marathons, protests, and other large public gatherings. Two strategies of applying pre-trained Imagenet network for event verification are presented, with two modifications for each strategy. The first method uses the features from the last convolutional layer of a pre-trained network as input to a classifier. We also consider the effects of tuning the convolutional weights of the pre-trained network to improve classification. The second method combines many features extracted from smaller scales and uses the output of a pre-trained network as the input to a second classifier. For both methods, we investigated several different classifiers and tested many different pre-trained networks. Our experiments demonstrate both these approaches are effective for event verification and image re-purposing detection. The classification at the global scale tends to marginally outperform our tested local methods and fine tuning the network further improves the results.