Learn to Sing by Listening: Building Controllable Virtual Singer by Unsupervised Learning from Voice Recordings

TL;DR

This paper introduces an unsupervised variational auto-encoder framework that digitizes and controls virtual singing voices by learning from voice recordings, enabling creation and manipulation of virtual singers without labeled data.

Contribution

It presents a novel unsupervised VAE-based approach to model and control singing voices using only speech recordings, eliminating the need for extensive labeled datasets.

Findings

Effective digitization of singing voices from speech recordings

Ability to control and interpolate vocal characteristics

Successful generation of virtual singers with controllable features

Abstract

The virtual world is being established in which digital humans are created indistinguishable from real humans. Producing their audio-related capabilities is crucial since voice conveys extensive personal characteristics. We aim to create a controllable audio-form virtual singer; however, supervised modeling and controlling all different factors of the singing voice, such as timbre, tempo, pitch, and lyrics, is extremely difficult since accurately labeling all such information needs enormous labor work. In this paper, we propose a framework that could digitize a person's voice by simply "listening" to the clean voice recordings of any content in a fully unsupervised manner and predict singing voices even only using speaking recordings. A variational auto-encoder (VAE) based framework is developed, which leverages a set of pre-trained models to encode the audio as various hidden embeddings representing different factors of the singing voice, and further decodes the embeddings into raw audio. By manipulating the hidden embeddings for different factors, the resulting singing voices can be controlled, and new virtual singers can also be further generated by interpolating between timbres. Evaluations of different types of experiments demonstrate the proposed method's effectiveness. The proposed method is the critical technique for producing the AI choir, which empowered the human-AI symbiotic orchestra in Hong Kong in July 2022.