Neurodyne: Neural Pitch Manipulation with Representation Learning and Cycle-Consistency GAN

TL;DR

Neurodyne introduces a neural pitch manipulation system that uses adversarial representation learning and cycle-consistency training to improve synthesis quality and maintain singer identity in music production.

Contribution

It presents a novel approach combining adversarial representation learning and cycle-consistency to enhance pitch manipulation without requiring paired data.

Findings

Improved synthesis quality in pitch manipulation tasks

Effective disentanglement of pitch-independent features

Maintains singer identity during pitch adjustments

Abstract

Pitch manipulation is the process of producers adjusting the pitch of an audio segment to a specific key and intonation, which is essential in music production. Neural-network-based pitch-manipulation systems have been popular in recent years due to their superior synthesis quality compared to classical DSP methods. However, their performance is still limited due to their inaccurate feature disentanglement using source-filter models and the lack of paired in- and out-of-tune training data. This work proposes Neurodyne to address these issues. Specifically, Neurodyne uses adversarial representation learning to learn a pitch-independent latent representation to avoid inaccurate disentanglement and cycle-consistency training to create paired training data implicitly. Experimental results on global-key and template-based pitch manipulation demonstrate the effectiveness of the proposed system, marking improved synthesis quality while maintaining the original singer identity.