Self-Supervised Contrastive Learning for Robust Audio-Sheet Music Retrieval Systems

TL;DR

This paper explores self-supervised contrastive learning to improve cross-modal music retrieval by pre-training on real music data, significantly enhancing snippet retrieval and piece identification accuracy in audio-sheet music systems.

Contribution

It demonstrates that self-supervised contrastive pre-training on real music data improves cross-modal retrieval performance and generalizes better to real-world scenarios.

Findings

Pre-trained models show better retrieval precision across scenarios.

Retrieval quality improves from 30% to 100% with real data.

Self-supervised learning alleviates data scarcity in music retrieval.

Abstract

Linking sheet music images to audio recordings remains a key problem for the development of efficient cross-modal music retrieval systems. One of the fundamental approaches toward this task is to learn a cross-modal embedding space via deep neural networks that is able to connect short snippets of audio and sheet music. However, the scarcity of annotated data from real musical content affects the capability of such methods to generalize to real retrieval scenarios. In this work, we investigate whether we can mitigate this limitation with self-supervised contrastive learning, by exposing a network to a large amount of real music data as a pre-training step, by contrasting randomly augmented views of snippets of both modalities, namely audio and sheet images. Through a number of experiments on synthetic and real piano data, we show that pre-trained models are able to retrieve snippets with better precision in all scenarios and pre-training configurations. Encouraged by these results, we employ the snippet embeddings in the higher-level task of cross-modal piece identification and conduct more experiments on several retrieval configurations. In this task, we observe that the retrieval quality improves from 30% up to 100% when real music data is present. We then conclude by arguing for the potential of self-supervised contrastive learning for alleviating the annotated data scarcity in multi-modal music retrieval models.