Self-supervised restoration of singing voice degraded by pitch shifting using shallow diffusion

TL;DR

This paper introduces a self-supervised, shallow diffusion model for high-quality singing voice restoration after pitch shifting, effectively reducing artifacts and preserving natural sound compared to traditional methods.

Contribution

It presents a novel self-supervised learning approach using a lightweight diffusion model in mel space for artifact-resistant pitch shifting in singing voices.

Findings

Significantly reduces pitch shift artifacts compared to classical baselines.

Uses self-supervised training with pitch shift reversal to simulate realistic artifacts.

Achieves natural sound restoration while preserving melody and timing.

Abstract

Pitch shifting has been an essential feature in singing voice production. However, conventional signal processing approaches exhibit well known trade offs such as formant shifts and robotic coloration that becomes more severe at larger transposition jumps. This paper targets high quality pitch shifting for singing by reframing it as a restoration problem: given an audio track that has been pitch shifted (and thus contaminated by artifacts), we recover a natural sounding performance while preserving its melody and timing. Specifically, we use a lightweight, mel space diffusion model driven by frame level acoustic features such as f0, volume, and content features. We construct training pairs in a self supervised manner by applying pitch shifts and reversing them to simulate realistic artifacts while retaining ground truth. On a curated singing set, the proposed approach substantially reduces pitch shift artifacts compared to representative classical baselines, as measured by both statistical metrics and pairwise acoustic measures. The results suggest that restoration based pitch shifting could be a viable approach towards artifact resistant transposition in vocal production workflows.