Evaluating Deep Music Generation Methods Using Data Augmentation

TL;DR

This paper proposes an objective framework to evaluate deep music generation models by measuring their impact on music mood classification performance, focusing on the meaningful information in generated samples rather than musical quality.

Contribution

It introduces a novel, homogeneous evaluation method based on classifier performance changes, and applies it to compare three deep music generation models for the first time.

Findings

Generated samples improve mood classification accuracy.

Models produce emotionally meaningful music.

Generated data validity is confirmed by classifier evaluation.

Abstract

Despite advances in deep algorithmic music generation, evaluation of generated samples often relies on human evaluation, which is subjective and costly. We focus on designing a homogeneous, objective framework for evaluating samples of algorithmically generated music. Any engineered measures to evaluate generated music typically attempt to define the samples' musicality, but do not capture qualities of music such as theme or mood. We do not seek to assess the musical merit of generated music, but instead explore whether generated samples contain meaningful information pertaining to emotion or mood/theme. We achieve this by measuring the change in predictive performance of a music mood/theme classifier after augmenting its training data with generated samples. We analyse music samples generated by three models -- SampleRNN, Jukebox, and DDSP -- and employ a homogeneous framework across all methods to allow for objective comparison. This is the first attempt at augmenting a music genre classification dataset with conditionally generated music. We investigate the classification performance improvement using deep music generation and the ability of the generators to make emotional music by using an additional, emotion annotation of the dataset. Finally, we use a classifier trained on real data to evaluate the label validity of class-conditionally generated samples.