Seeing Your Speech Style: A Novel Zero-Shot Identity-Disentanglement Face-based Voice Conversion

TL;DR

This paper introduces ID-FaceVC, a novel face-based voice conversion method that effectively disentangles speaker identity from content, allowing high-quality, controllable voice synthesis from facial images, audio, or text inputs.

Contribution

The paper proposes a new identity-disentanglement framework with contrastive learning and mutual information modules, enabling improved voice conversion and controllable speech generation.

Findings

Achieves state-of-the-art performance in voice conversion metrics

Effectively disentangles speaker identity from content

Supports controllable speech generation with emotional tone and speed

Abstract

Face-based Voice Conversion (FVC) is a novel task that leverages facial images to generate the target speaker's voice style. Previous work has two shortcomings: (1) suffering from obtaining facial embeddings that are well-aligned with the speaker's voice identity information, and (2) inadequacy in decoupling content and speaker identity information from the audio input. To address these issues, we present a novel FVC method, Identity-Disentanglement Face-based Voice Conversion (ID-FaceVC), which overcomes the above two limitations. More precisely, we propose an Identity-Aware Query-based Contrastive Learning (IAQ-CL) module to extract speaker-specific facial features, and a Mutual Information-based Dual Decoupling (MIDD) module to purify content features from audio, ensuring clear and high-quality voice conversion. Besides, unlike prior works, our method can accept either audio or text inputs, offering controllable speech generation with adjustable emotional tone and speed. Extensive experiments demonstrate that ID-FaceVC achieves state-of-the-art performance across various metrics, with qualitative and user study results confirming its effectiveness in naturalness, similarity, and diversity. Project website with audio samples and code can be found at https://id-facevc.github.io.