Learning to Groove with Inverse Sequence Transformations

TL;DR

This paper introduces a novel approach for translating musical ideas into expressive performances by adapting sequence-to-sequence models and GAN-inspired transformations, demonstrating applications in drumming and performance humanization.

Contribution

It adapts GAN-based sequence transformation techniques to music, creates a new drumming dataset, and improves performance humanization methods.

Findings

Successfully adapted GAN-inspired sequence transformations for music.

Created and released a new dataset of professional drumming recordings.

Achieved improvements over state-of-the-art humanization methods.

Abstract

We explore models for translating abstract musical ideas (scores, rhythms) into expressive performances using Seq2Seq and recurrent Variational Information Bottleneck (VIB) models. Though Seq2Seq models usually require painstakingly aligned corpora, we show that it is possible to adapt an approach from the Generative Adversarial Network (GAN) literature (e.g. Pix2Pix (Isola et al., 2017) and Vid2Vid (Wang et al. 2018a)) to sequences, creating large volumes of paired data by performing simple transformations and training generative models to plausibly invert these transformations. Music, and drumming in particular, provides a strong test case for this approach because many common transformations (quantization, removing voices) have clear semantics, and models for learning to invert them have real-world applications. Focusing on the case of drum set players, we create and release a new dataset for this purpose, containing over 13 hours of recordings by professional drummers aligned with fine-grained timing and dynamics information. We also explore some of the creative potential of these models, including demonstrating improvements on state-of-the-art methods for Humanization (instantiating a performance from a musical score).