Towards Source Attribution of Singing Voice Deepfake with Multimodal Foundation Models

TL;DR

This paper introduces a new task for identifying the source of singing voice deepfakes using multimodal foundation models, demonstrating their effectiveness and proposing a novel fusion framework for improved accuracy.

Contribution

The paper proposes the SVDSA task, evaluates multimodal foundation models for it, and introduces COFFE, a fusion framework with a novel Chernoff Distance-based loss function.

Findings

MMFMs outperform speech and music models in SVDSA.

Fusion of foundation models improves attribution accuracy.

COFFE achieves state-of-the-art performance in source attribution.

Abstract

In this work, we introduce the task of singing voice deepfake source attribution (SVDSA). We hypothesize that multimodal foundation models (MMFMs) such as ImageBind, LanguageBind will be most effective for SVDSA as they are better equipped for capturing subtle source-specific characteristics-such as unique timbre, pitch manipulation, or synthesis artifacts of each singing voice deepfake source due to their cross-modality pre-training. Our experiments with MMFMs, speech foundation models and music foundation models verify the hypothesis that MMFMs are the most effective for SVDSA. Furthermore, inspired from related research, we also explore fusion of foundation models (FMs) for improved SVDSA. To this end, we propose a novel framework, COFFE which employs Chernoff Distance as novel loss function for effective fusion of FMs. Through COFFE with the symphony of MMFMs, we attain the topmost performance in comparison to all the individual FMs and baseline fusion methods.