SinTra: Learning an inspiration model from a single multi-track music segment

TL;DR

SinTra is a novel auto-regressive model that learns to generate coherent, multi-instrument polyphonic music from a single segment using a pyramid Transformer-XL architecture and a new pitch-group representation.

Contribution

The paper introduces SinTra, a single-segment learning framework with a pyramid Transformer-XL and a pitch-group representation for high-quality multi-instrument music generation.

Findings

SinTra outperforms Music Transformer in learning from a single music segment.

The pyramid structure reduces overly-fragmented notes.

The model effectively captures musical structure and inter-track relationships.

Abstract

In this paper, we propose SinTra, an auto-regressive sequential generative model that can learn from a single multi-track music segment, to generate coherent, aesthetic, and variable polyphonic music of multi-instruments with an arbitrary length of bar. For this task, to ensure the relevance of generated samples and training music, we present a novel pitch-group representation. SinTra, consisting of a pyramid of Transformer-XL with a multi-scale training strategy, can learn both the musical structure and the relative positional relationship between notes of the single training music segment. Additionally, for maintaining the inter-track correlation, we use the convolution operation to process multi-track music, and when decoding, the tracks are independent to each other to prevent interference. We evaluate SinTra with both subjective study and objective metrics. The comparison results show that our framework can learn information from a single music segment more sufficiently than Music Transformer. Also the comparison between SinTra and its variant, i.e., the single-stage SinTra with the first stage only, shows that the pyramid structure can effectively suppress overly-fragmented notes.