Universal Speech Content Factorization

TL;DR

Universal Speech Content Factorization (USCF) is a linear method that effectively isolates phonetic content from speaker identity in speech, enabling zero-shot voice conversion and efficient speech synthesis without extensive target data.

Contribution

USCF extends existing speech content factorization to an open-set setting using a universal mapping, enabling zero-shot voice conversion with minimal target speech and serving as a timbre-disentangled feature for TTS.

Findings

USCF effectively removes speaker-dependent variation.

USCF achieves competitive zero-shot voice conversion quality.

USCF features improve training efficiency for timbre-prompted TTS.

Abstract

We propose Universal Speech Content Factorization (USCF), a simple and invertible linear method for extracting a low-rank speech representation in which speaker timbre is suppressed while phonetic content is preserved. USCF extends Speech Content Factorization, a closed-set voice conversion (VC) method, to an open-set setting by learning a universal speech-to-content mapping via least-squares optimization and deriving speaker-specific transformations from only a few seconds of target speech. We show through embedding analysis that USCF effectively removes speaker-dependent variation. As a zero-shot VC system, USCF achieves competitive intelligibility, naturalness, and speaker similarity compared to methods that require substantially more target-speaker data or additional neural training. Finally, we demonstrate that as a training-efficient timbre-disentangled speech feature, USCF features can serve as the acoustic representation for training timbre-prompted text-to-speech models. Speech samples and code are publicly available.