Diffusion Timbre Transfer Via Mutual Information Guided Inpainting

TL;DR

This paper presents a novel inference-time method for timbre transfer in music audio using a pre-trained latent diffusion model, involving noise injection and structural clamping to control instrument style while preserving musical structure.

Contribution

It introduces a lightweight, training-free approach for timbre transfer that leverages mutual information-guided inpainting on audio latents, compatible with text/audio conditioning.

Findings

Effective timbre transfer with structural preservation

Inference-time controls enable style steering

Compatible with text/audio conditioning models

Abstract

We study timbre transfer as an inference-time editing problem for music audio. Starting from a strong pre-trained latent diffusion model, we introduce a lightweight procedure that requires no additional training: (i) a dimension-wise noise injection that targets latent channels most informative of instrument identity, and (ii) an early-step clamping mechanism that re-imposes the input's melodic and rhythmic structure during reverse diffusion. The method operates directly on audio latents and is compatible with text/audio conditioning (e.g., CLAP). We discuss design choices,analyze trade-offs between timbral change and structural preservation, and show that simple inference-time controls can meaningfully steer pre-trained models for style-transfer use cases.