KaraTuner: Towards end to end natural pitch correction for singing voice in karaoke

TL;DR

KaraTuner is an end-to-end neural system that automatically corrects pitch in singing voices for karaoke, ensuring naturalness and sound quality through integrated pitch prediction and vocoding.

Contribution

It introduces a novel neural architecture combining pitch prediction and vocoding for seamless, end-to-end pitch correction in singing voices.

Findings

Outperforms rule-based pitch correction in preference tests.

Achieves higher timbre consistency and sound quality than existing vocoders.

Demonstrates effective long-term dependency modeling with Transformer.

Abstract

An automatic pitch correction system typically includes several stages, such as pitch extraction, deviation estimation, pitch shift processing, and cross-fade smoothing. However, designing these components with strategies often requires domain expertise and they are likely to fail on corner cases. In this paper, we present KaraTuner, an end-to-end neural architecture that predicts pitch curve and resynthesizes the singing voice directly from the tuned pitch and vocal spectrum extracted from the original recordings. Several vital technical points have been introduced in KaraTuner to ensure pitch accuracy, pitch naturalness, timbre consistency, and sound quality. A feed-forward Transformer is employed in the pitch predictor to capture longterm dependencies in the vocal spectrum and musical note. We also develop a pitch-controllable vocoder based on a novel source-filter block and the Fre-GAN architecture. KaraTuner obtains a higher preference than the rule-based pitch correction approach through A/B tests, and perceptual experiments show that the proposed vocoder achieves significant advantages in timbre consistency and sound quality compared with the parametric WORLD vocoder, phase vocoder and CLPC vocoder.