Revealing the Truth with ConLLM for Detecting Multi-Modal Deepfakes

TL;DR

This paper introduces ConLLM, a hybrid framework utilizing contrastive learning and large language models to improve the detection of multi-modal deepfakes by addressing modality fragmentation and semantic inconsistency issues.

Contribution

ConLLM is a novel two-stage approach that combines pre-trained models and LLM-based reasoning to enhance multi-modal deepfake detection performance.

Findings

Reduces audio deepfake EER by up to 50%

Improves video detection accuracy by up to 8%

Achieves about 9% accuracy gains in audio-visual deepfake detection

Abstract

The rapid rise of deepfake technology poses a severe threat to social and political stability by enabling hyper-realistic synthetic media capable of manipulating public perception. However, existing detection methods struggle with two core limitations: (1) modality fragmentation, which leads to poor generalization across diverse and adversarial deepfake modalities; and (2) shallow inter-modal reasoning, resulting in limited detection of fine-grained semantic inconsistencies. To address these, we propose ConLLM (Contrastive Learning with Large Language Models), a hybrid framework for robust multimodal deepfake detection. ConLLM employs a two-stage architecture: stage 1 uses Pre-Trained Models (PTMs) to extract modality-specific embeddings; stage 2 aligns these embeddings via contrastive learning to mitigate modality fragmentation, and refines them using LLM-based reasoning to address shallow inter-modal reasoning by capturing semantic inconsistencies. ConLLM demonstrates strong performance across audio, video, and audio-visual modalities. It reduces audio deepfake EER by up to 50%, improves video accuracy by up to 8%, and achieves approximately 9% accuracy gains in audio-visual tasks. Ablation studies confirm that PTM-based embeddings contribute 9%-10% consistent improvements across modalities.