GSRM: Generative Speech Reward Model for Speech RLHF

TL;DR

This paper introduces GSRM, a novel generative reward model for speech naturalness evaluation that offers interpretability and improves upon existing predictors, enhancing speech synthesis quality.

Contribution

The paper proposes GSRM, a reasoning-centric, interpretable reward model for speech naturalness, trained with large-scale human feedback and benchmarked against existing methods.

Findings

GSRM outperforms existing speech naturalness predictors.

GSRM achieves model-human correlation approaching inter-rater consistency.

GSRM improves speech LLM generation naturalness via RLHF verification.

Abstract

Recent advances in speech language models, such as GPT-4o Voice Mode and Gemini Live, have demonstrated promising speech generation capabilities. Nevertheless, the aesthetic naturalness of the synthesized audio still lags behind that of human speech. Enhancing generation quality requires a reliable evaluator of speech naturalness. However, existing naturalness evaluators typically regress raw audio to scalar scores, offering limited interpretability of the evaluation and moreover fail to generalize to speech across different taxonomies. Inspired by recent advances in generative reward modeling, we propose the Generative Speech Reward Model (GSRM), a reasoning-centric reward model tailored for speech. The GSRM is trained to decompose speech naturalness evaluation into an interpretable acoustic feature extraction stage followed by feature-grounded chain-of-thought reasoning, enabling explainable judgments. To achieve this, we curated a large-scale human feedback dataset comprising 31k expert ratings and an out-of-domain benchmark of real-world user-assistant speech interactions. Experiments show that GSRM substantially outperforms existing speech naturalness predictors, achieving model-human correlation of naturalness score prediction that approaches human inter-rater consistency. We further show how GSRM can improve the naturalness of speech LLM generations by serving as an effective verifier for online RLHF.