ImprovNet -- Generating Controllable Musical Improvisations with Iterative Corruption Refinement

TL;DR

ImprovNet is a transformer-based model that enables controllable, expressive musical improvisations and style transfer across genres, using an iterative corruption-refinement approach to modify melody, harmony, or rhythm while maintaining musical coherence.

Contribution

It introduces a unified transformer architecture with a self-supervised training strategy for controllable, multi-task musical style transfer and improvisation, outperforming previous models in coherence and genre recognition.

Findings

Outperforms Anticipatory Music Transformer in short tasks

Achieves 79% genre recognition accuracy in style transfer

Effectively controls style transfer degree and structural similarity

Abstract

Despite deep learning's remarkable advances in style transfer across various domains, generating controllable performance-level musical style transfer for complete symbolically represented musical works remains a challenging area of research. Much of this is owed to limited datasets, especially for genres such as jazz, and the lack of unified models that can handle multiple music generation tasks. This paper presents ImprovNet, a transformer-based architecture that generates expressive and controllable musical improvisations through a self-supervised corruption-refinement training strategy. The improvisational style transfer is aimed at making meaningful modifications to one or more musical elements - melody, harmony or rhythm of the original composition with respect to the target genre. ImprovNet unifies multiple capabilities within a single model: it can perform cross-genre and intra-genre improvisations, harmonize melodies with genre-specific styles, and execute short prompt continuation and infilling tasks. The model's iterative generation framework allows users to control the degree of style transfer and structural similarity to the original composition. Objective and subjective evaluations demonstrate ImprovNet's effectiveness in generating musically coherent improvisations while maintaining structural relationships with the original pieces. The model outperforms Anticipatory Music Transformer in short continuation and infilling tasks and successfully achieves recognizable genre conversion, with 79\% of participants correctly identifying jazz-style improvisations of classical pieces. Our code and demo page can be found at https://github.com/keshavbhandari/improvnet.