LADLE-MM: Limited Annotation based Detector with Learned Ensembles for Multimodal Misinformation

TL;DR

LADLE-MM is a resource-efficient multimodal misinformation detector that performs well with limited annotations, leveraging learned ensembles and fixed multimodal embeddings to outperform more complex models.

Contribution

The paper introduces LADLE-MM, a novel multimodal misinformation detection model that requires fewer trainable parameters and limited annotations, yet achieves competitive and superior performance.

Findings

Achieves competitive results with 60.3% fewer trainable parameters.

Outperforms state-of-the-art models on DGM4 and VERITE datasets.

Demonstrates strong robustness and generalization in open-set scenarios.

Abstract

With the rise of easily accessible tools for generating and manipulating multimedia content, realistic synthetic alterations to digital media have become a widespread threat, often involving manipulations across multiple modalities simultaneously. Recently, such techniques have been increasingly employed to distort narratives of important events and to spread misinformation on social media, prompting the development of misinformation detectors. In the context of misinformation conveyed through image-text pairs, several detection methods have been proposed. However, these approaches typically rely on computationally intensive architectures or require large amounts of annotated data. In this work we introduce LADLE-MM: Limited Annotation based Detector with Learned Ensembles for Multimodal Misinformation, a model-soup initialized multimodal misinformation detector designed to operate under a limited annotation setup and constrained training resources. LADLE-MM is composed of two unimodal branches and a third multimodal one that enhances image and text representations with additional multimodal embeddings extracted from BLIP, serving as fixed reference space. Despite using 60.3% fewer trainable parameters than previous state-of-the-art models, LADLE-MM achieves competitive performance on both binary and multi-label classification tasks on the DGM4 benchmark, outperforming existing methods when trained without grounding annotations. Moreover, when evaluated on the VERITE dataset, LADLE-MM outperforms current state-of-the-art approaches that utilize more complex architectures involving Large Vision-Language-Models, demonstrating the effective generalization ability in an open-set setting and strong robustness to unimodal bias.