$\text{M}^3\text{PDB}$: A Multimodal, Multi-Label, Multilingual Prompt Database for Speech Generation

TL;DR

This paper introduces $ ext{M}^3 ext{PDB}$, a large-scale, multilingual, multi-modal prompt database designed to improve zero-shot speech generation in real-world scenarios with diverse and imperfect prompts.

Contribution

We present the first multi-modal, multi-label, multilingual prompt database for speech generation and a prompt selection strategy optimized for real-time, resource-constrained environments.

Findings

The database supports diverse speech generation scenarios.

The prompt selection strategy improves robustness in challenging conditions.

Experimental results validate the effectiveness of the dataset and method.

Abstract

Recent advancements in zero-shot speech generation have enabled models to synthesize speech that mimics speaker identity and speaking style from speech prompts. However, these models' effectiveness is significantly limited in real-world scenarios where high-quality speech prompts are absent, incomplete, or out of domain. This issue arises primarily from a significant quality mismatch between the speech data utilized for model training and the input prompt speech during inference. To address this, we introduce $\text{M}^3\text{PDB}$, the first large-scale, multi-modal, multi-label, and multilingual prompt database designed for robust prompt selection in speech generation. Our dataset construction leverages a novel multi-modal, multi-agent annotation framework, enabling precise and hierarchical labeling across diverse modalities. Furthermore, we propose a lightweight yet effective prompt selection strategy tailored for real-time, resource-constrained inference settings. Experimental results demonstrate that our proposed database and selection strategy effectively support various challenging speech generation scenarios. We hope our work can inspire the community to shift focus from improving performance on standard benchmarks to addressing more realistic and diverse application scenarios in speech generation. Code and dataset are available at: https://github.com/hizening/M3PDB.