Conan: A Chunkwise Online Network for Zero-Shot Adaptive Voice Conversion

TL;DR

Conan is a novel online zero-shot voice conversion model that effectively preserves content, adapts to unseen speakers, and produces natural speech in real-time, addressing key limitations of existing VC systems.

Contribution

We propose Conan, a chunkwise online VC model with a streaming content extractor, style encoder, and causal vocoder, enabling real-time zero-shot voice conversion.

Findings

Outperforms baseline models in subjective quality

Achieves low-latency real-time conversion

Effectively adapts to unseen speaker styles

Abstract

Zero-shot online voice conversion (VC) holds significant promise for real-time communications and entertainment. However, current VC models struggle to preserve semantic fidelity under real-time constraints, deliver natural-sounding conversions, and adapt effectively to unseen speaker characteristics. To address these challenges, we introduce Conan, a chunkwise online zero-shot voice conversion model that preserves the content of the source while matching the voice timbre and styles of reference speech. Conan comprises three core components: 1) a Stream Content Extractor that leverages Emformer for low-latency streaming content encoding; 2) an Adaptive Style Encoder that extracts fine-grained stylistic features from reference speech for enhanced style adaptation; 3) a Causal Shuffle Vocoder that implements a fully causal HiFiGAN using a pixel-shuffle mechanism. Experimental evaluations demonstrate that Conan outperforms baseline models in subjective and objective metrics. Audio samples can be found at https://aaronz345.github.io/ConanDemo.