Multi-Reward GRPO for Stable and Prosodic Single-Codebook TTS LLMs at Scale

TL;DR

This paper introduces a multi-reward reinforcement learning framework to improve prosody, stability, and naturalness in single-codebook TTS large language models, addressing common issues like prosody instability and speaker drift.

Contribution

It proposes a novel multi-reward GRPO method that directly optimizes token generation for better prosody and stability, incorporating rule-based rewards and external LLM annotations.

Findings

Enhanced prosodic stability and naturalness in TTS models.

Consistent improvements across different data sizes and model scales.

Additional gains when attaching a flow-matching decoder.

Abstract

Recent advances in Large Language Models (LLMs) have transformed text-to-speech (TTS) synthesis, inspiring autoregressive frameworks that represent speech as sequences of discrete codec tokens. Among them, single-codebook TTS LLMs have emerged as compact and streamable architectures that jointly model semantic and acoustic integration. However, despite their efficiency, these models often exhibit unstable prosody, speaker drift, and degraded naturalness. To address these issues, we propose a multi-reward Group Relative Policy Optimization (GRPO) framework that directly optimizes the token generation policy of single-codebook TTS LLMs. Beyond standard intelligibility and speaker similarity objectives, our design integrates three rule-based rewards: a length penalty for duration consistency, an entropy regularization reward for decoding stability, and an LLM-annotated prosody alignment reward that explicitly supervises rhythm. In this prosody reward, an external reasoning LLM predicts multiple plausible pause structures via in-context learning, providing a human-preference-aligned supervisory signal for GRPO training. To assess universality, we further attach a flow-matching (FM) decoder on top of the GRPO-optimized AR backbone and observe consistent additional gains, indicating that our reinforcement optimization enhances the intrinsic AR policy. We further conduct a scalability analysis across data sizes and model scales, revealing that the proposed method consistently enhances prosodic stability, speaker similarity, and overall speech naturalness in single-codebook TTS LLMs.