InstructME: An Instruction Guided Music Edit And Remix Framework with Latent Diffusion Models

TL;DR

InstructME is a novel instruction-guided music editing framework using latent diffusion models that enhances musical harmony and coherence during editing and remixing tasks.

Contribution

It introduces a multi-scale U-Net, chord progression conditioning, and a chunk transformer to improve music editing quality and handle long-term dependencies.

Findings

Outperforms previous methods in music quality and harmony

Effective in instrument editing and remixing tasks

Supports multi-round editing with consistent results

Abstract

Music editing primarily entails the modification of instrument tracks or remixing in the whole, which offers a novel reinterpretation of the original piece through a series of operations. These music processing methods hold immense potential across various applications but demand substantial expertise. Prior methodologies, although effective for image and audio modifications, falter when directly applied to music. This is attributed to music's distinctive data nature, where such methods can inadvertently compromise the intrinsic harmony and coherence of music. In this paper, we develop InstructME, an Instruction guided Music Editing and remixing framework based on latent diffusion models. Our framework fortifies the U-Net with multi-scale aggregation in order to maintain consistency before and after editing. In addition, we introduce chord progression matrix as condition information and incorporate it in the semantic space to improve melodic harmony while editing. For accommodating extended musical pieces, InstructME employs a chunk transformer, enabling it to discern long-term temporal dependencies within music sequences. We tested InstructME in instrument-editing, remixing, and multi-round editing. Both subjective and objective evaluations indicate that our proposed method significantly surpasses preceding systems in music quality, text relevance and harmony. Demo samples are available at https://musicedit.github.io/