Are Music Foundation Models Better at Singing Voice Deepfake Detection? Far-Better Fuse them with Speech Foundation Models

TL;DR

This paper compares music and speech foundation models for singing voice deepfake detection, finding speech models, especially speaker recognition ones, perform best, and introduces FIONA, a fusion framework that achieves state-of-the-art results.

Contribution

It is the first comprehensive comparison of MFMs and SFMs for SVDD and proposes a novel fusion framework, FIONA, to improve detection performance.

Findings

Speaker recognition SFMs outperform other models in SVDD.

Fusion of FMs via FIONA improves detection accuracy.

FIONA achieves the lowest EER of 13.74%, surpassing previous methods.

Abstract

In this study, for the first time, we extensively investigate whether music foundation models (MFMs) or speech foundation models (SFMs) work better for singing voice deepfake detection (SVDD), which has recently attracted attention in the research community. For this, we perform a comprehensive comparative study of state-of-the-art (SOTA) MFMs (MERT variants and music2vec) and SFMs (pre-trained for general speech representation learning as well as speaker recognition). We show that speaker recognition SFM representations perform the best amongst all the foundation models (FMs), and this performance can be attributed to its higher efficacy in capturing the pitch, tone, intensity, etc, characteristics present in singing voices. To our end, we also explore the fusion of FMs for exploiting their complementary behavior for improved SVDD, and we propose a novel framework, FIONA for the same. With FIONA, through the synchronization of x-vector (speaker recognition SFM) and MERT-v1-330M (MFM), we report the best performance with the lowest Equal Error Rate (EER) of 13.74 %, beating all the individual FMs as well as baseline FM fusions and achieving SOTA results.