Parameter-Efficient Transfer Learning for Music Foundation Models

TL;DR

This paper explores parameter-efficient transfer learning methods for music foundation models, demonstrating they outperform probing and fine-tuning in music auto-tagging and match fine-tuning in key detection and tempo estimation with less training cost.

Contribution

It introduces three PETL methods—adapter, prompt, and reparameterization-based—that require fewer parameters and computational resources for adapting music foundation models.

Findings

PETL methods outperform probing and fine-tuning in music auto-tagging.

PETL methods achieve similar results to fine-tuning in key detection and tempo estimation with less training.

Current foundation models' usefulness for key and tempo tasks is questioned due to small models performing similarly.

Abstract

More music foundation models are recently being released, promising a general, mostly task independent encoding of musical information. Common ways of adapting music foundation models to downstream tasks are probing and fine-tuning. These common transfer learning approaches, however, face challenges. Probing might lead to suboptimal performance because the pre-trained weights are frozen, while fine-tuning is computationally expensive and is prone to overfitting. Our work investigates the use of parameter-efficient transfer learning (PETL) for music foundation models which integrates the advantage of probing and fine-tuning. We introduce three types of PETL methods: adapter-based methods, prompt-based methods, and reparameterization-based methods. These methods train only a small number of parameters, and therefore do not require significant computational resources. Results show that PETL methods outperform both probing and fine-tuning on music auto-tagging. On key detection and tempo estimation, they achieve similar results as fine-tuning with significantly less training cost. However, the usefulness of the current generation of foundation model on key and tempo tasks is questioned by the similar results achieved by training a small model from scratch. Code available at https://github.com/suncerock/peft-music/