MIDI-VALLE: Improving Expressive Piano Performance Synthesis Through Neural Codec Language Modelling

TL;DR

MIDI-VALLE is a neural codec language model that improves expressive piano performance synthesis by encoding MIDI and audio as discrete tokens, enabling better generalisation and synthesis quality across diverse styles and recordings.

Contribution

The paper introduces MIDI-VALLE, a novel neural codec model that conditions on reference audio and MIDI, encoding performances as discrete tokens for enhanced synthesis and generalisation.

Findings

Achieves over 75% lower Frechet Audio Distance than baseline.

Receives significantly more votes in listening tests, indicating higher perceived quality.

Demonstrates superior performance across diverse datasets.

Abstract

Generating expressive audio performances from music scores requires models to capture both instrument acoustics and human interpretation. Traditional music performance synthesis pipelines follow a two-stage approach, first generating expressive performance MIDI from a score, then synthesising the MIDI into audio. However, the synthesis models often struggle to generalise across diverse MIDI sources, musical styles, and recording environments. To address these challenges, we propose MIDI-VALLE, a neural codec language model adapted from the VALLE framework, which was originally designed for zero-shot personalised text-to-speech (TTS) synthesis. For performance MIDI-to-audio synthesis, we improve the architecture to condition on a reference audio performance and its corresponding MIDI. Unlike previous TTS-based systems that rely on piano rolls, MIDI-VALLE encodes both MIDI and audio as discrete tokens, facilitating a more consistent and robust modelling of piano performances. Furthermore, the model's generalisation ability is enhanced by training on an extensive and diverse piano performance dataset. Evaluation results show that MIDI-VALLE significantly outperforms a state-of-the-art baseline, achieving over 75% lower Frechet Audio Distance on the ATEPP and Maestro datasets. In the listening test, MIDI-VALLE received 202 votes compared to 58 for the baseline, demonstrating improved synthesis quality and generalisation across diverse performance MIDI inputs.