Phoneme-Level Deepfake Detection Across Emotional Conditions Using Self-Supervised Embeddings

TL;DR

This paper introduces a phoneme-level analysis framework using self-supervised embeddings to improve detection of emotionally manipulated synthetic speech, enhancing interpretability and robustness.

Contribution

It proposes a novel phoneme-level approach with WavLM embeddings for detecting emotional deepfakes, addressing limitations of previous homogeneous signal methods.

Findings

Complex vowels and fricatives show higher divergence in synthetic speech.

Phonemes with larger distributional differences are more detectable.

Phoneme-level analysis improves interpretability in deepfake detection.

Abstract

Recent advances in emotional voice conversion (EVC) have enabled the generation of expressive synthetic speech, raising new concerns in audio deepfake detection. Existing approaches treat speech as a homogeneous signal and largely overlook its internal phonetic structure, limiting their interpretability in emotionally conditioned settings. In this work, we propose a phoneme-level framework to analyze emotionally manipulated synthetic speech using real and EVC-generated speech under matched emotional conditions with shared transcripts, phoneme-aligned TextGrids, and WavLM-based embeddings. Our results show that phoneme behavior varies across categories, with complex vowels and fricatives exhibiting higher divergence while simpler phonemes remain more stable. Phonemes with larger distributional differences are also found to be more easily detected, consistently across multiple emotions and synthesis systems. These findings demonstrate that phoneme-level analysis is an effective and interpretable approach for detecting emotionally manipulated synthetic speech.