Two Views, One Truth: Spectral and Self-Supervised Features Fusion for Robust Speech Deepfake Detection

TL;DR

This paper proposes hybrid fusion methods combining self-supervised and spectral features to improve the robustness and generalization of speech deepfake detection, outperforming single-modality approaches across multiple benchmarks.

Contribution

It introduces several fusion strategies, including cross attention and gating mechanisms, to effectively combine waveform and spectral features for deepfake detection.

Findings

Fusion approaches outperform SSL-only baseline.

Cross attention achieves 38% relative EER reduction.

Joint modeling enhances robustness and generalization.

Abstract

Recent advances in synthetic speech have made audio deepfakes increasingly realistic, posing significant security risks. Existing detection methods that rely on a single modality, either raw waveform embeddings or spectral based features, are vulnerable to non spoof disturbances and often overfit to known forgery algorithms, resulting in poor generalization to unseen attacks. To address these shortcomings, we investigate hybrid fusion frameworks that integrate self supervised learning (SSL) based representations with handcrafted spectral descriptors (MFCC , LFCC, CQCC). By aligning and combining complementary information across modalities, these fusion approaches capture subtle artifacts that single feature approaches typically overlook. We explore several fusion strategies, including simple concatenation, cross attention, mutual cross attention, and a learnable gating mechanism, to optimally blend SSL features with fine grained spectral cues. We evaluate our approach on four challenging public benchmarks and report generalization performance. All fusion variants consistently outperform an SSL only baseline, with the cross attention strategy achieving the best generalization with a 38% relative reduction in equal error rate (EER). These results confirm that joint modeling of waveform and spectral views produces robust, domain agnostic representations for audio deepfake detection.