Investigating the Viability of Employing Multi-modal Large Language Models in the Context of Audio Deepfake Detection

TL;DR

This paper explores the potential of multimodal large language models for audio deepfake detection, demonstrating promising results with in-domain data but highlighting challenges in generalization and zero-shot performance.

Contribution

It introduces the use of multimodal large language models with question-answer prompts for audio deepfake detection, a novel approach in this domain.

Findings

Models perform well on in-domain data with minimal supervision.

Zero-shot performance is poor without task-specific training.

Generalization to out-of-domain data remains a challenge.

Abstract

While Vision-Language Models (VLMs) and Multimodal Large Language Models (MLLMs) have shown strong generalisation in detecting image and video deepfakes, their use for audio deepfake detection remains largely unexplored. In this work, we aim to explore the potential of MLLMs for audio deepfake detection. Combining audio inputs with a range of text prompts as queries to find out the viability of MLLMs to learn robust representations across modalities for audio deepfake detection. Therefore, we attempt to explore text-aware and context-rich, question-answer based prompts with binary decisions. We hypothesise that such a feature-guided reasoning will help in facilitating deeper multimodal understanding and enable robust feature learning for audio deepfake detection. We evaluate the performance of two MLLMs, Qwen2-Audio-7B-Instruct and SALMONN, in two evaluation modes: (a) zero-shot and (b) fine-tuned. Our experiments demonstrate that combining audio with a multi-prompt approach could be a viable way forward for audio deepfake detection. Our experiments show that the models perform poorly without task-specific training and struggle to generalise to out-of-domain data. However, they achieve good performance on in-domain data with minimal supervision, indicating promising potential for audio deepfake detection.