CoCoEmo: Composable and Controllable Human-Like Emotional TTS via Activation Steering

TL;DR

This paper introduces a novel activation steering framework for hybrid TTS models that enables controllable, composable, and mixed-emotion speech synthesis, revealing that emotional prosody is primarily generated by the language module.

Contribution

It systematically analyzes emotion control via activation steering in hybrid TTS, proposing a quantitative framework and evaluation protocols for mixed-emotion synthesis.

Findings

Emotional prosody mainly generated by the language module.

Lightweight steering approach produces natural emotional speech.

Framework enables reliable synthesis of mixed and mismatched emotions.

Abstract

Emotional expression in human speech is nuanced and compositional, often involving multiple, sometimes conflicting, affective cues that may diverge from linguistic content. In contrast, most expressive text-to-speech systems enforce a single utterance-level emotion, collapsing affective diversity and suppressing mixed or text-emotion-misaligned expression. While activation steering via latent direction vectors offers a promising solution, it remains unclear whether emotion representations are linearly steerable in TTS, where steering should be applied within hybrid TTS architectures, and how such complex emotion behaviors should be evaluated. This paper presents the first systematic analysis of activation steering for emotional control in hybrid TTS models, introducing a quantitative, controllable steering framework, and multi-rater evaluation protocols that enable composable mixed-emotion synthesis and reliable text-emotion mismatch synthesis. Our results demonstrate, for the first time, that emotional prosody and expressive variability are primarily synthesized by the TTS language module instead of the flow-matching module, and also provide a lightweight steering approach for generating natural, human-like emotional speech.