RAVE for Speech: Efficient Voice Conversion at High Sampling Rates

TL;DR

This paper introduces RAVE for Speech, a high-sampling-rate voice conversion method that emphasizes model simplicity and efficiency, achieving naturalness comparable to state-of-the-art while reducing inference time.

Contribution

It proposes a novel time-domain voice conversion approach using speech representation learning and latent space guidance for high-quality, efficient conversion.

Findings

Achieves comparable naturalness and quality to state-of-the-art methods.

Significantly reduces inference time.

Struggles with similarity to unseen speakers.

Abstract

Voice conversion has gained increasing popularity within the field of audio manipulation and speech synthesis. Often, the main objective is to transfer the input identity to that of a target speaker without changing its linguistic content. While current work provides high-fidelity solutions they rarely focus on model simplicity, high-sampling rate environments or stream-ability. By incorporating speech representation learning into a generative timbre transfer model, traditionally created for musical purposes, we investigate the realm of voice conversion generated directly in the time domain at high sampling rates. More specifically, we guide the latent space of a baseline model towards linguistically relevant representations and condition it on external speaker information. Through objective and subjective assessments, we demonstrate that the proposed solution can attain levels of naturalness, quality, and intelligibility comparable to those of a state-of-the-art solution for seen speakers, while significantly decreasing inference time. However, despite the presence of target speaker characteristics in the converted output, the actual similarity to unseen speakers remains a challenge.