Unispeaker: A Unified Approach for Multimodality-driven Speaker Generation

TL;DR

UniSpeaker introduces a unified multimodal voice generation framework that maps diverse voice descriptions into a shared space, improving alignment and outperforming previous models across multiple tasks.

Contribution

It proposes a novel KV-Former based voice aggregator and establishes the first multimodality-based voice control benchmark.

Findings

Outperforms previous modality-specific models

Effective voice alignment with soft contrastive loss

Validated across five diverse tasks

Abstract

Recent advancements in personalized speech generation have brought synthetic speech increasingly close to the realism of target speakers' recordings, yet multimodal speaker generation remains on the rise. This paper introduces UniSpeaker, a unified approach for multimodality-driven speaker generation. Specifically, we propose a unified voice aggregator based on KV-Former, applying soft contrastive loss to map diverse voice description modalities into a shared voice space, ensuring that the generated voice aligns more closely with the input descriptions. To evaluate multimodality-driven voice control, we build the first multimodality-based voice control (MVC) benchmark, focusing on voice suitability, voice diversity, and speech quality. UniSpeaker is evaluated across five tasks using the MVC benchmark, and the experimental results demonstrate that UniSpeaker outperforms previous modality-specific models. Speech samples are available at \url{https://UniSpeaker.github.io}.