MIDI-LLM: Adapting Large Language Models for Text-to-MIDI Music Generation

TL;DR

MIDI-LLM is a novel large language model adapted for generating multitrack MIDI music from text prompts, combining vocabulary expansion and a two-stage training process for improved quality and control.

Contribution

It introduces a method to adapt existing LLMs for text-to-MIDI music generation by expanding vocabulary and training strategies, enabling faster inference and better performance.

Findings

Achieves higher quality music generation than recent models

Provides better text control over generated music

Enables faster inference using existing LLM infrastructure

Abstract

We present MIDI-LLM, an LLM for generating multitrack MIDI music from free-form text prompts. Our approach expands a text LLM's vocabulary to include MIDI tokens, and uses a two-stage training recipe to endow text-to-MIDI abilities. By preserving the original LLM's parameter structure, we can directly leverage the vLLM library for accelerated inference. Experiments show that MIDI-LLM achieves higher quality, better text control, and faster inference compared to the recent Text2midi model. Live demo at https://midi-llm-demo.vercel.app.