Towards Emotionally Consistent Text-Based Speech Editing: Introducing EmoCorrector and The ECD-TSE Dataset

TL;DR

This paper introduces EmoCorrector, a post-correction method for text-based speech editing that improves emotional consistency, supported by a new dataset, ECD-TSE, and validated through experiments showing enhanced emotional expression.

Contribution

The paper presents EmoCorrector, a novel retrieval-augmented generation approach for emotional correction in TSE, and introduces the ECD-TSE dataset for benchmarking emotional consistency.

Findings

EmoCorrector significantly improves emotional expression in synthetic speech.

ECD-TSE dataset enables comprehensive evaluation of emotional consistency.

Experiments show enhanced alignment of speech emotion with text edits.

Abstract

Text-based speech editing (TSE) modifies speech using only text, eliminating re-recording. However, existing TSE methods, mainly focus on the content accuracy and acoustic consistency of synthetic speech segments, and often overlook the emotional shifts or inconsistency issues introduced by text changes. To address this issue, we propose EmoCorrector, a novel post-correction scheme for TSE. EmoCorrector leverages Retrieval-Augmented Generation (RAG) by extracting the edited text's emotional features, retrieving speech samples with matching emotions, and synthesizing speech that aligns with the desired emotion while preserving the speaker's identity and quality. To support the training and evaluation of emotional consistency modeling in TSE, we pioneer the benchmarking Emotion Correction Dataset for TSE (ECD-TSE). The prominent aspect of ECD-TSE is its inclusion of $<$text, speech$>$ paired data featuring diverse text variations and a range of emotional expressions. Subjective and objective experiments and comprehensive analysis on ECD-TSE confirm that EmoCorrector significantly enhances the expression of intended emotion while addressing emotion inconsistency limitations in current TSE methods. Code and audio examples are available at https://github.com/AI-S2-Lab/EmoCorrector.