Nested Music Transformer: Sequentially Decoding Compound Tokens in Symbolic Music and Audio Generation

TL;DR

The paper introduces the Nested Music Transformer, an autoregressive model that decodes compound tokens in symbolic music and audio, improving sequence modeling efficiency and performance by capturing sub-token interdependencies.

Contribution

It presents a novel nested transformer architecture for autoregressive decoding of compound tokens, reducing memory usage while enhancing modeling of sub-token relationships.

Findings

Improved perplexity on symbolic music datasets

Enhanced modeling of sub-token interdependencies

Efficient decoding of compound tokens in audio and music

Abstract

Representing symbolic music with compound tokens, where each token consists of several different sub-tokens representing a distinct musical feature or attribute, offers the advantage of reducing sequence length. While previous research has validated the efficacy of compound tokens in music sequence modeling, predicting all sub-tokens simultaneously can lead to suboptimal results as it may not fully capture the interdependencies between them. We introduce the Nested Music Transformer (NMT), an architecture tailored for decoding compound tokens autoregressively, similar to processing flattened tokens, but with low memory usage. The NMT consists of two transformers: the main decoder that models a sequence of compound tokens and the sub-decoder for modeling sub-tokens of each compound token. The experiment results showed that applying the NMT to compound tokens can enhance the performance in terms of better perplexity in processing various symbolic music datasets and discrete audio tokens from the MAESTRO dataset.