A Unified Neural Codec Language Model for Selective Editable Text to Speech Generation

TL;DR

This paper introduces SpeechEdit, a neural codec language model that enables zero-shot TTS with selective attribute control, allowing for flexible and localized speech attribute editing while maintaining naturalness.

Contribution

The paper presents a novel unified codec language model with a selective control mechanism for zero-shot TTS, trained on a new dataset for attribute-specific speech editing.

Findings

Maintains naturalness and robustness in controlled TTS.

Enables selective attribute editing in speech synthesis.

Achieves flexible, localized control over speech attributes.

Abstract

Neural codec language models achieve impressive zero-shot Text-to-Speech (TTS) by fully imitating the acoustic characteristics of a short speech prompt, including timbre, prosody, and paralinguistic information. However, such holistic imitation limits their ability to isolate and control individual attributes. In this paper, we present a unified codec language model SpeechEdit that extends zero-shot TTS with a selective control mechanism. By default, SpeechEdit reproduces the complete acoustic profile inferred from the speech prompt, but it selectively overrides only the attributes specified by explicit control instructions. To enable controllable modeling, SpeechEdit is trained on our newly constructed LibriEdit dataset, which provides delta (difference-aware) training pairs derived from LibriHeavy. Experimental results show that our approach maintains naturalness and robustness while offering flexible and localized control over desired attributes. Audio samples are available at https://speech-editing.github.io/speech-editing/.