Music Generation based on Generative Adversarial Networks with Transformer

TL;DR

This paper introduces a GAN-based approach with Transformer models for music generation, addressing long-sequence quality issues by combining adversarial training with likelihood methods, resulting in improved sample quality.

Contribution

It proposes integrating a pre-trained Span-BERT discriminator and Gumbel-Softmax for stable adversarial training of Transformer-based music models, enhancing long-sequence generation quality.

Findings

Outperforms likelihood-only models in human evaluations

Uses a Span-BERT discriminator for stable GAN training

Introduces a novel discriminative metric for assessment

Abstract

Autoregressive models based on Transformers have become the prevailing approach for generating music compositions that exhibit comprehensive musical structure. These models are typically trained by minimizing the negative log-likelihood (NLL) of the observed sequence in an autoregressive manner. However, when generating long sequences, the quality of samples from these models tends to significantly deteriorate due to exposure bias. To address this issue, we leverage classifiers trained to differentiate between real and sampled sequences to identify these failures. This observation motivates our exploration of adversarial losses as a complement to the NLL objective. We employ a pre-trained Span-BERT model as the discriminator in the Generative Adversarial Network (GAN) framework, which enhances training stability in our experiments. To optimize discrete sequences within the GAN framework, we utilize the Gumbel-Softmax trick to obtain a differentiable approximation of the sampling process. Additionally, we partition the sequences into smaller chunks to ensure that memory constraints are met. Through human evaluations and the introduction of a novel discriminative metric, we demonstrate that our approach outperforms a baseline model trained solely on likelihood maximization.