CoVLM: Leveraging Consensus from Vision-Language Models for Semi-supervised Multi-modal Fake News Detection

TL;DR

This paper introduces CoVLM, a semi-supervised framework that leverages vision-language models to detect out-of-context fake news by generating pseudo-labels from unlabeled image-text pairs, reducing reliance on extensive labeled data.

Contribution

The paper presents a novel semi-supervised method that automatically determines thresholds for pseudo-labeling in fake news detection, improving performance with limited labeled data.

Findings

Effective pseudo-label generation for unlabeled data

Improved detection accuracy over state-of-the-art methods

Robustness across challenging datasets

Abstract

In this work, we address the real-world, challenging task of out-of-context misinformation detection, where a real image is paired with an incorrect caption for creating fake news. Existing approaches for this task assume the availability of large amounts of labeled data, which is often impractical in real-world, since it requires extensive manual intervention and domain expertise. In contrast, since obtaining a large corpus of unlabeled image-text pairs is much easier, here, we propose a semi-supervised protocol, where the model has access to a limited number of labeled image-text pairs and a large corpus of unlabeled pairs. Additionally, the occurrence of fake news being much lesser compared to the real ones, the datasets tend to be highly imbalanced, thus making the task even more challenging. Towards this goal, we propose a novel framework, Consensus from Vision-Language Models (CoVLM), which generates robust pseudo-labels for unlabeled pairs using thresholds derived from the labeled data. This approach can automatically determine the right threshold parameters of the model for selecting the confident pseudo-labels. Experimental results on benchmark datasets across challenging conditions and comparisons with state-of-the-art approaches demonstrate the effectiveness of our framework.