WaveTransfer: A Flexible End-to-end Multi-instrument Timbre Transfer with Diffusion

TL;DR

WaveTransfer is a versatile diffusion-based model for high-quality, multi-instrument timbre transfer in music, capable of handling various instrument pairs and different sampling rates without retraining.

Contribution

It introduces WaveTransfer, a novel end-to-end diffusion model that performs multi-instrument timbre transfer at multiple sampling rates, including 44.1 kHz, with a single trained model.

Findings

Successfully transfers timbre between multiple instrument pairs.

Operates effectively at 44.1 kHz sampling rate.

Handles both individual instrument and mixture timbre transfer.

Abstract

As diffusion-based deep generative models gain prevalence, researchers are actively investigating their potential applications across various domains, including music synthesis and style alteration. Within this work, we are interested in timbre transfer, a process that involves seamlessly altering the instrumental characteristics of musical pieces while preserving essential musical elements. This paper introduces WaveTransfer, an end-to-end diffusion model designed for timbre transfer. We specifically employ the bilateral denoising diffusion model (BDDM) for noise scheduling search. Our model is capable of conducting timbre transfer between audio mixtures as well as individual instruments. Notably, it exhibits versatility in that it accommodates multiple types of timbre transfer between unique instrument pairs in a single model, eliminating the need for separate model training for each pairing. Furthermore, unlike recent works limited to 16 kHz, WaveTransfer can be trained at various sampling rates, including the industry-standard 44.1 kHz, a feature of particular interest to the music community.