CosyEdit: Unlocking End-to-End Speech Editing Capability from Zero-Shot Text-to-Speech Models

TL;DR

CosyEdit is an end-to-end speech editing model that leverages fine-tuned zero-shot TTS models to perform high-quality speech modifications efficiently, outperforming large language model baselines and matching state-of-the-art cascade methods.

Contribution

We introduce CosyEdit, a novel end-to-end speech editing approach that internalizes speech-text alignment, enabling effective editing from a zero-shot TTS model with minimal supervised data.

Findings

Outperforms several billion-parameter language model baselines.

Matches the performance of state-of-the-art cascade approaches.

Achieves high consistency between original and edited speech.

Abstract

Automatic speech editing aims to modify spoken content based on textual instructions, yet traditional cascade systems suffer from complex preprocessing pipelines and a reliance on explicit external temporal alignment. Addressing these limitations, we propose CosyEdit, an end-to-end speech editing model adapted from CosyVoice through task-specific fine-tuning and an optimized inference procedure, which internalizes speech-text alignment while ensuring high consistency between the speech before and after editing. By fine-tuning on only 250 hours of supervised data from our curated GigaEdit dataset, our 400M-parameter model achieves reliable speech editing performance. Experiments on the RealEdit benchmark indicate that CosyEdit not only outperforms several billion-parameter language model baselines but also matches the performance of state-of-the-art cascade approaches. These results demonstrate that, with task-specific fine-tuning and inference optimization, robust and efficient speech editing capabilities can be unlocked from a zero-shot TTS model, yielding a novel and cost-effective end-to-end solution for high-quality speech editing.