TVTSyn: Content-Synchronous Time-Varying Timbre for Streaming Voice Conversion and Anonymization

TL;DR

TVTSyn introduces a content-synchronous, time-varying timbre representation for streaming voice conversion and anonymization, enabling low-latency, natural, and privacy-preserving speech synthesis with improved speaker transfer and intelligibility.

Contribution

It proposes a novel streamable speech synthesizer with a time-varying timbre representation that aligns content and identity at the frame level, enhancing privacy and naturalness.

Findings

Achieves <80 ms GPU latency in streaming synthesis.

Outperforms state-of-the-art baselines in naturalness and speaker transfer.

Effectively reduces residual speaker leakage through vector-quantized regularization.

Abstract

Real-time voice conversion and speaker anonymization require causal, low-latency synthesis without sacrificing intelligibility or naturalness. Current systems have a core representational mismatch: content is time-varying, while speaker identity is injected as a static global embedding. We introduce a streamable speech synthesizer that aligns the temporal granularity of identity and content via a content-synchronous, time-varying timbre (TVT) representation. A Global Timbre Memory expands a global timbre instance into multiple compact facets; frame-level content attends to this memory, a gate regulates variation, and spherical interpolation preserves identity geometry while enabling smooth local changes. In addition, a factorized vector-quantized bottleneck regularizes content to reduce residual speaker leakage. The resulting system is streamable end-to-end, with <80 ms GPU latency. Experiments show improvements in naturalness, speaker transfer, and anonymization compared to SOTA streaming baselines, establishing TVT as a scalable approach for privacy-preserving and expressive speech synthesis under strict latency budgets.