RRPO: Robust Reward Policy Optimization for LLM-based Emotional TTS

TL;DR

This paper introduces RRPO, a new framework for emotion-controlled text-to-speech that enhances robustness against reward hacking, leading to more natural and expressive speech synthesis aligned with human perception.

Contribution

RRPO employs a hybrid regularization scheme to develop a robust reward model that mitigates reward hacking and improves emotional expressiveness in TTS.

Findings

Robust RM generalizes well across languages.

RRPO significantly improves emotional expressiveness.

Mitigates reward hacking effectively.

Abstract

Differentiable reinforcement learning (RL) frameworks like DiffRO offer a powerful approach for controllable text-to-speech (TTS), but are vulnerable to reward hacking, particularly for nuanced tasks like emotion control. The policy model can exploit a vanilla Reward Model (RM) by generating acoustic artifacts to achieve spurious rewards, but at the cost of degrading perceptual quality. To address this, we propose Robust Reward Policy Optimization (RRPO), a novel framework that employs a hybrid regularization scheme. This scheme develops a robust RM whose reward signal is more reliably aligned with human perception, compelling the policy to abandon detrimental shortcuts and instead learn the complex features of genuine emotions. Our ablation study confirms the enhanced robustness of our RM, as evidenced by its strong cross-lingual generalization. The subjective evaluation demonstrates that this robust RM effectively mitigates reward hacking, leading to significant improvements in both emotional expressiveness and naturalness over all baselines. Demo page: https://lrwinr.github.io/RRPO-CosyVoice.