Large Visual-Language Models Are Also Good Classifiers: A Study of In-Context Multimodal Fake News Detection

TL;DR

This study evaluates large visual-language models' ability to detect fake news, demonstrating that integrating smaller model predictions into in-context learning significantly improves their accuracy in multimodal fake news detection tasks.

Contribution

The paper introduces the IMFND framework that combines in-context learning with predictions from smaller models, substantially enhancing LVLMs' fake news detection performance.

Findings

LVLMs perform competitively with smaller models in zero-shot FND.

In-context learning improves FND performance but is limited without additional strategies.

IMFND framework significantly boosts LVLMs' accuracy across multiple datasets.

Abstract

Large visual-language models (LVLMs) exhibit exceptional performance in visual-language reasoning across diverse cross-modal benchmarks. Despite these advances, recent research indicates that Large Language Models (LLMs), like GPT-3.5-turbo, underachieve compared to well-trained smaller models, such as BERT, in Fake News Detection (FND), prompting inquiries into LVLMs' efficacy in FND tasks. Although performance could improve through fine-tuning LVLMs, the substantial parameters and requisite pre-trained weights render it a resource-heavy endeavor for FND applications. This paper initially assesses the FND capabilities of two notable LVLMs, CogVLM and GPT4V, in comparison to a smaller yet adeptly trained CLIP model in a zero-shot context. The findings demonstrate that LVLMs can attain performance competitive with that of the smaller model. Next, we integrate standard in-context learning (ICL) with LVLMs, noting improvements in FND performance, though limited in scope and consistency. To address this, we introduce the \textbf{I}n-context \textbf{M}ultimodal \textbf{F}ake \textbf{N}ews \textbf{D}etection (IMFND) framework, enriching in-context examples and test inputs with predictions and corresponding probabilities from a well-trained smaller model. This strategic integration directs the LVLMs' focus towards news segments associated with higher probabilities, thereby improving their analytical accuracy. The experimental results suggest that the IMFND framework significantly boosts the FND efficiency of LVLMs, achieving enhanced accuracy over the standard ICL approach across three publicly available FND datasets.