Leveraging large multimodal models for audio-video deepfake detection: a pilot study

TL;DR

This paper introduces AV-LMMDetect, a large multimodal model fine-tuned for audio-visual deepfake detection, which outperforms prior models and demonstrates strong generalization across datasets.

Contribution

The paper presents AV-LMMDetect, a large multimodal model that effectively detects deepfakes by jointly analyzing audio and visual streams, improving upon existing small, task-specific detectors.

Findings

Matches or surpasses prior methods on FakeAVCeleb and Mavos-DD datasets.

Sets a new state of the art on Mavos-DD datasets.

Demonstrates strong cross-domain generalization.

Abstract

Audio-visual deepfake detection (AVD) is increasingly important as modern generators can fabricate convincing speech and video. Most current multimodal detectors are small, task-specific models: they work well on curated tests but scale poorly and generalize weakly across domains. We introduce AV-LMMDetect, a supervised fine-tuned (SFT) large multimodal model that casts AVD as a prompted yes/no classification - "Is this video real or fake?". Built on Qwen 2.5 Omni, it jointly analyzes audio and visual streams for deepfake detection and is trained in two stages: lightweight LoRA alignment followed by audio-visual encoder full fine-tuning. On FakeAVCeleb and Mavos-DD, AV-LMMDetect matches or surpasses prior methods and sets a new state of the art on Mavos-DD datasets.