KLASSify to Verify: Audio-Visual Deepfake Detection Using SSL-based Audio and Handcrafted Visual Features

TL;DR

This paper introduces a multimodal deepfake detection method combining handcrafted visual features and SSL-based audio representations, achieving high accuracy and robustness against unseen deepfake attacks.

Contribution

It proposes a novel multimodal approach utilizing handcrafted visual features and SSL-based audio models for improved deepfake detection and localization.

Findings

Achieved 92.78% AUC on deepfake classification

Attained 0.3536 IoU for temporal localization using audio

Balances detection performance with real-world applicability

Abstract

The rapid development of audio-driven talking head generators and advanced Text-To-Speech (TTS) models has led to more sophisticated temporal deepfakes. These advances highlight the need for robust methods capable of detecting and localizing deepfakes, even under novel, unseen attack scenarios. Current state-of-the-art deepfake detectors, while accurate, are often computationally expensive and struggle to generalize to novel manipulation techniques. To address these challenges, we propose multimodal approaches for the AV-Deepfake1M 2025 challenge. For the visual modality, we leverage handcrafted features to improve interpretability and adaptability. For the audio modality, we adapt a self-supervised learning (SSL) backbone coupled with graph attention networks to capture rich audio representations, improving detection robustness. Our approach strikes a balance between performance and real-world deployment, focusing on resilience and potential interpretability. On the AV-Deepfake1M++ dataset, our multimodal system achieves AUC of 92.78% for deepfake classification task and IoU of 0.3536 for temporal localization using only the audio modality.