Pictures Of MIDI: Controlled Music Generation via Graphical Prompts for Image-Based Diffusion Inpainting

TL;DR

This paper introduces a graphical interface for controlled music inpainting using an Hourglass Diffusion Transformer trained on MIDI images, enabling intuitive user control and enhanced note generation without latent space autoencoders.

Contribution

It presents a novel, efficient pixel-space diffusion model with graphical masking controls for music inpainting, improving user interaction and structural diversity in generated music.

Findings

Effective inpainting of melodies and accompaniments.

Enhanced note density and structural control through repainting.

Operates efficiently at longer context windows without autoencoders.

Abstract

Recent years have witnessed significant progress in generative models for music, featuring diverse architectures that balance output quality, diversity, speed, and user control. This study explores a user-friendly graphical interface enabling the drawing of masked regions for inpainting by an Hourglass Diffusion Transformer (HDiT) model trained on MIDI piano roll images. To enhance note generation in specified areas, masked regions can be "repainted" with extra noise. The non-latent HDiTs linear scaling with pixel count allows efficient generation in pixel space, providing intuitive and interpretable controls such as masking throughout the network and removing the need to operate in compressed latent spaces such as those provided by pretrained autoencoders. We demonstrate that, in addition to inpainting of melodies, accompaniment, and continuations, the use of repainting can help increase note density yielding musical structures closely matching user specifications such as rising, falling, or diverging melody and/or accompaniment, even when these lie outside the typical training data distribution. We achieve performance on par with prior results while operating at longer context windows, with no autoencoder, and can enable complex geometries for inpainting masks, increasing the options for machine-assisted composers to control the generated music.