How does fake news use a thumbnail? CLIP-based Multimodal Detection on the Unrepresentative News Image
Hyewon Choi, Yejun Yoon, Seunghyun Yoon, Kunwoo Park

TL;DR
This paper explores how fake news uses misleading thumbnails by leveraging CLIP-based multimodal analysis to detect semantic incongruities between images and news text, aiding misinformation detection.
Contribution
It introduces a novel CLIP-based approach to identify semantic mismatches between news images and text, enhancing fake news detection methods.
Findings
Fake news employs more incongruous images compared to general news.
CLIP-based methods effectively detect image-text incongruity in news articles.
The approach provides a new perspective on combating online misinformation.
Abstract
This study investigates how fake news uses a thumbnail for a news article with a focus on whether a news article's thumbnail represents the news content correctly. A news article shared with an irrelevant thumbnail can mislead readers into having a wrong impression of the issue, especially in social media environments where users are less likely to click the link and consume the entire content. We propose to capture the degree of semantic incongruity in the multimodal relation by using the pretrained CLIP representation. From a source-level analysis, we found that fake news employs a more incongruous image to the main content than general news. Going further, we attempted to detect news articles with image-text incongruity. Evaluation experiments suggest that CLIP-based methods can successfully detect news articles in which the thumbnail is semantically irrelevant to news text. This…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMisinformation and Its Impacts · Topic Modeling · Authorship Attribution and Profiling
MethodsContrastive Language-Image Pre-training
