LinearVC: Linear transformations of self-supervised features through the lens of voice conversion

TL;DR

LinearVC demonstrates that simple linear transformations of self-supervised speech features can effectively perform voice conversion, revealing that content and speaker information are linearly separable in a low-dimensional space.

Contribution

The paper introduces LinearVC, a straightforward method for voice conversion using linear transformations, and provides insights into the structure of self-supervised speech representations.

Findings

Linear transformations, especially rotations, suffice for high-quality voice conversion.

Content information resides in a low-dimensional subspace that can be linearly manipulated.

A rank-100 linear projection achieves competitive voice conversion results.

Abstract

We introduce LinearVC, a simple voice conversion method that sheds light on the structure of self-supervised representations. First, we show that simple linear transformations of self-supervised features effectively convert voices. Next, we probe the geometry of the feature space by constraining the set of allowed transformations. We find that just rotating the features is sufficient for high-quality voice conversion. This suggests that content information is embedded in a low-dimensional subspace which can be linearly transformed to produce a target voice. To validate this hypothesis, we finally propose a method that explicitly factorizes content and speaker information using singular value decomposition; the resulting linear projection with a rank of just 100 gives competitive conversion results. Our work has implications for both practical voice conversion and a broader understanding of self-supervised speech representations. Samples and code: https://www.kamperh.com/linearvc/.