Advanced Zero-Shot Text-to-Speech for Background Removal and Preservation with Controllable Masked Speech Prediction

TL;DR

This paper introduces a novel zero-shot text-to-speech method that allows precise control over background removal or preservation, enhancing naturalness and contextual integrity in diverse acoustic environments.

Contribution

It proposes a controllable masked speech prediction strategy with a dual-speaker encoder, enabling flexible background handling in zero-shot TTS systems.

Findings

Effective background removal and preservation across various conditions

Strong generalization to unseen acoustic scenarios

Precise control over background handling demonstrated

Abstract

The acoustic background plays a crucial role in natural conversation. It provides context and helps listeners understand the environment, but a strong background makes it difficult for listeners to understand spoken words. The appropriate handling of these backgrounds is situation-dependent: Although it may be necessary to remove background to ensure speech clarity, preserving the background is sometimes crucial to maintaining the contextual integrity of the speech. Despite recent advancements in zero-shot Text-to-Speech technologies, current systems often struggle with speech prompts containing backgrounds. To address these challenges, we propose a Controllable Masked Speech Prediction strategy coupled with a dual-speaker encoder, utilizing a task-related control signal to guide the prediction of dual background removal and preservation targets. Experimental results demonstrate that our approach enables precise control over the removal or preservation of background across various acoustic conditions and exhibits strong generalization capabilities in unseen scenarios.