Annotation-Free MIDI-to-Audio Synthesis via Concatenative Synthesis and Generative Refinement

TL;DR

CoSaRef is a novel MIDI-to-audio synthesis method that generates diverse, realistic instrumental tracks without requiring MIDI annotations, using concatenative synthesis and diffusion-based refinement for fine-grained control.

Contribution

It introduces a MIDI-to-audio synthesis approach that eliminates the need for MIDI annotations, enhancing diversity and control in generated audio.

Findings

Outperforms state-of-the-art MIDI-supervised methods in quality and control.

Enables detailed timbre and expression control via audio samples and MIDI design.

Produces realistic, expressive tracks with diverse instrument timbres.

Abstract

Recent MIDI-to-audio synthesis methods using deep neural networks have successfully generated high-quality, expressive instrumental tracks. However, these methods require MIDI annotations for supervised training, limiting the diversity of instrument timbres and expression styles in the output. We propose CoSaRef, a MIDI-to-audio synthesis method that does not require MIDI-audio paired datasets. CoSaRef first generates a synthetic audio track using concatenative synthesis based on MIDI input, then refines it with a diffusion-based deep generative model trained on datasets without MIDI annotations. This approach improves the diversity of timbres and expression styles. Additionally, it allows detailed control over timbres and expression through audio sample selection and extra MIDI design, similar to traditional functions in digital audio workstations. Experiments showed that CoSaRef could generate realistic tracks while preserving fine-grained timbre control via one-shot samples. Moreover, despite not being supervised on MIDI annotation, CoSaRef outperformed the state-of-the-art timbre-controllable method based on MIDI supervision in both objective and subjective evaluation.