Evaluation of CNN-based Automatic Music Tagging Models

TL;DR

This paper provides a consistent evaluation of CNN-based music tagging models across multiple datasets, analyzing their robustness to input perturbations and offering reproducible implementations for future research.

Contribution

It offers a standardized comparison framework for CNN music tagging models and assesses their generalization under various input perturbations.

Findings

Models achieve comparable performance on standard metrics.

Perturbations reduce model accuracy, indicating sensitivity.

Reproducible pre-trained models are provided for future research.

Abstract

Recent advances in deep learning accelerated the development of content-based automatic music tagging systems. Music information retrieval (MIR) researchers proposed various architecture designs, mainly based on convolutional neural networks (CNNs), that achieve state-of-the-art results in this multi-label binary classification task. However, due to the differences in experimental setups followed by researchers, such as using different dataset splits and software versions for evaluation, it is difficult to compare the proposed architectures directly with each other. To facilitate further research, in this paper we conduct a consistent evaluation of different music tagging models on three datasets (MagnaTagATune, Million Song Dataset, and MTG-Jamendo) and provide reference results using common evaluation metrics (ROC-AUC and PR-AUC). Furthermore, all the models are evaluated with perturbed inputs to investigate the generalization capabilities concerning time stretch, pitch shift, dynamic range compression, and addition of white noise. For reproducibility, we provide the PyTorch implementations with the pre-trained models.