Source Tracing of Synthetic Speech Systems Through Paralinguistic Pre-Trained Representations

TL;DR

This paper introduces TRIO, a novel framework that fuses paralinguistic and speaker recognition pre-trained speech representations to improve source tracing of synthetic speech systems, achieving state-of-the-art results.

Contribution

It proposes a new fusion method combining paralinguistic and speaker recognition models with a gated mechanism and CCA loss for better source attribution in synthetic speech.

Findings

TRIO outperforms individual SPTMs and baseline fusion methods.

Fusing TRILLsson and x-vector improves source tracing accuracy.

The approach sets new state-of-the-art in synthetic speech source tracing.

Abstract

In this work, we focus on source tracing of synthetic speech generation systems (STSGS). Each source embeds distinctive paralinguistic features--such as pitch, tone, rhythm, and intonation--into their synthesized speech, reflecting the underlying design of the generation model. While previous research has explored representations from speech pre-trained models (SPTMs), the use of representations from SPTM pre-trained for paralinguistic speech processing, which excel in paralinguistic tasks like synthetic speech detection, speech emotion recognition has not been investigated for STSGS. We hypothesize that representations from paralinguistic SPTM will be more effective due to its ability to capture source-specific paralinguistic cues attributing to its paralinguistic pre-training. Our comparative study of representations from various SOTA SPTMs, including paralinguistic, monolingual, multilingual, and speaker recognition, validates this hypothesis. Furthermore, we explore fusion of representations and propose TRIO, a novel framework that fuses SPTMs using a gated mechanism for adaptive weighting, followed by canonical correlation loss for inter-representation alignment and self-attention for feature refinement. By fusing TRILLsson (Paralinguistic SPTM) and x-vector (Speaker recognition SPTM), TRIO outperforms individual SPTMs, baseline fusion methods, and sets new SOTA for STSGS in comparison to previous works.