Zero-Shot Voice Conditioning for Denoising Diffusion TTS Models

TL;DR

This paper introduces a zero-shot voice conditioning technique for denoising diffusion TTS models, enabling the generation of speech in a new speaker's voice using only a short sample, without additional training.

Contribution

It proposes a novel inference-time conditioning method that combines denoising diffusion with a low-pass filtered speaker sample, eliminating the need for retraining.

Findings

Achieves speaker similarity comparable to state-of-the-art methods

Requires only a 3-second sample from the target speaker

Operates without any additional training steps

Abstract

We present a novel way of conditioning a pretrained denoising diffusion speech model to produce speech in the voice of a novel person unseen during training. The method requires a short (~3 seconds) sample from the target person, and generation is steered at inference time, without any training steps. At the heart of the method lies a sampling process that combines the estimation of the denoising model with a low-pass version of the new speaker's sample. The objective and subjective evaluations show that our sampling method can generate a voice similar to that of the target speaker in terms of frequency, with an accuracy comparable to state-of-the-art methods, and without training.