AudioMorphix: Training-free audio editing with diffusion probabilistic models

TL;DR

AudioMorphix is a training-free, diffusion-based audio editing method that enables precise, localized modifications to spectrograms by referencing other recordings, preserving audio fidelity and supporting diverse editing tasks.

Contribution

It introduces a novel, training-free approach for localized audio editing using diffusion models and morphing theory, with an enhanced self-attention mechanism and a new evaluation benchmark.

Findings

Achieves high fidelity in various editing tasks

Enables precise modifications while preserving original audio quality

Demonstrates promising results across multiple editing scenarios

Abstract

Editing sound with precision is a crucial yet underexplored challenge in audio content creation. While existing works can manipulate sounds by text instructions or audio exemplar pairs, they often struggled to modify audio content precisely while preserving fidelity to the original recording. In this work, we introduce a novel editing approach that enables localized modifications to specific time-frequency regions while keeping the remaining of the audio intact by operating on spectrograms directly. To achieve this, we propose AudioMorphix, a training-free audio editor that manipulates a target region on the spectrogram by referring to another recording. Inspired by morphing theory, we conceptualize audio mixing as a process where different sounds blend seamlessly through morphing and can be decomposed back into individual components via demorphing. Our AudioMorphix optimizes the noised latent conditioned on raw input and reference audio while rectifying the guided diffusion process through a series of energy functions. Additionally, we enhance self-attention layers with a cache mechanism to preserve detailed characteristics from the original recordings. To advance audio editing research, we devise a new evaluation benchmark, which includes a curated dataset with a variety of editing instructions. Extensive experiments demonstrate that AudioMorphix yields promising performance on various audio editing tasks, including addition, removal, time shifting and stretching, and pitch shifting, achieving high fidelity and precision. Demo and code are available at this url.