How does fake news use a thumbnail? CLIP-based Multimodal Detection on the Unrepresentative News Image

TL;DR

This paper explores how fake news uses misleading thumbnails by leveraging CLIP-based multimodal analysis to detect semantic incongruities between images and news text, aiding misinformation detection.

Contribution

It introduces a novel CLIP-based approach to identify semantic mismatches between news images and text, enhancing fake news detection methods.

Findings

Fake news employs more incongruous images compared to general news.

CLIP-based methods effectively detect image-text incongruity in news articles.

The approach provides a new perspective on combating online misinformation.

Abstract

This study investigates how fake news uses a thumbnail for a news article with a focus on whether a news article's thumbnail represents the news content correctly. A news article shared with an irrelevant thumbnail can mislead readers into having a wrong impression of the issue, especially in social media environments where users are less likely to click the link and consume the entire content. We propose to capture the degree of semantic incongruity in the multimodal relation by using the pretrained CLIP representation. From a source-level analysis, we found that fake news employs a more incongruous image to the main content than general news. Going further, we attempted to detect news articles with image-text incongruity. Evaluation experiments suggest that CLIP-based methods can successfully detect news articles in which the thumbnail is semantically irrelevant to news text. This study contributes to the research by providing a novel view on tackling online fake news and misinformation. Code and datasets are available at https://github.com/ssu-humane/fake-news-thumbnail.