Voice Conversion With Just Nearest Neighbors

TL;DR

This paper introduces kNN-VC, a simple and effective voice conversion method that uses nearest neighbor search on self-supervised representations, achieving comparable speaker similarity to complex models with greater simplicity.

Contribution

The paper presents a novel, straightforward voice conversion approach using nearest neighbors on self-supervised features, simplifying the process while maintaining quality.

Findings

Improves speaker similarity over baseline methods

Maintains similar intelligibility scores

Simplifies the voice conversion pipeline

Abstract

Any-to-any voice conversion aims to transform source speech into a target voice with just a few examples of the target speaker as a reference. Recent methods produce convincing conversions, but at the cost of increased complexity -- making results difficult to reproduce and build on. Instead, we keep it simple. We propose k-nearest neighbors voice conversion (kNN-VC): a straightforward yet effective method for any-to-any conversion. First, we extract self-supervised representations of the source and reference speech. To convert to the target speaker, we replace each frame of the source representation with its nearest neighbor in the reference. Finally, a pretrained vocoder synthesizes audio from the converted representation. Objective and subjective evaluations show that kNN-VC improves speaker similarity with similar intelligibility scores to existing methods. Code, samples, trained models: https://bshall.github.io/knn-vc