RVCBench: Benchmarking the Robustness of Voice Cloning Across Modern Audio Generation Models

TL;DR

RVCBench is a comprehensive benchmark that evaluates the robustness of modern voice cloning models across various realistic challenges, revealing significant vulnerabilities and guiding future improvements.

Contribution

This paper introduces RVCBench, the first extensive benchmark for assessing robustness of voice cloning models under real-world conditions, covering multiple tasks and models.

Findings

Substantial robustness gaps in current VC models.

Performance drops under input shifts and post-processing.

Long-context and cross-lingual scenarios reveal stability issues.

Abstract

Modern voice cloning (VC) can synthesize speech that closely matches a target speaker from only seconds of reference audio, enabling applications such as personalized speech interfaces and dubbing. In practical deployments, modern audio generation models inevitably encounter noisy reference audios, imperfect text prompts, and diverse downstream processing, which can significantly hurt robustness. Despite rapid progress in VC driven by autoregressive codec-token language models and diffusion-based models, robustness under realistic deployment shifts remains underexplored. This paper introduces RVCBench, a comprehensive benchmark that evaluates Robustness in VC across the full generation pipeline, including input variation, generation challenges, output post-processing, and adversarial perturbations, covering 10 robustness tasks, 225 speakers, 14,370 utterances, and 11 representative modern VC models. Our evaluation uncovers substantial robustness gaps in VC: performance can deteriorate sharply under common input shifts and post-processing; long-context and cross-lingual scenarios further expose stability limitations; and both passive noise and proactive perturbation influence generation robustness. Collectively, these findings provide a unified picture of how current VC models fail in practice and introduce a standardized, open-source testbed to support the development of more robust and deployable VC models. We open-source our project at https://github.com/Nanboy-Ronan/RVCBench.