Exploiting Temporal Dependencies for Cross-Modal Music Piece Identification

TL;DR

This paper enhances cross-modal music identification by incorporating temporal alignment and attention mechanisms into deep learning models, significantly improving retrieval accuracy between audio recordings and sheet music images.

Contribution

It introduces temporal sequence alignment and an attention mechanism to improve cross-modal music retrieval, addressing limitations of previous embedding-based methods.

Findings

Significant improvement in piece and fragment retrieval accuracy

Effective handling of tempo variations with attention mechanisms

Scalability considerations for large-scale applications

Abstract

This paper addresses the problem of cross-modal musical piece identification and retrieval: finding the appropriate recording(s) from a database given a sheet music query, and vice versa, working directly with audio and scanned sheet music images. The fundamental approach to this is to learn a cross-modal embedding space with a suitable similarity structure for audio and sheet image snippets, using a deep neural network, and identifying candidate pieces by cross-modal near neighbour search in this space. However, this method is oblivious of temporal aspects of music. In this paper, we introduce two strategies that address this shortcoming. First, we present a strategy that aligns sequences of embeddings learned from sheet music scans and audio snippets. A series of experiments on whole piece and fragment-level retrieval on 24 hours worth of classical piano recordings demonstrates significant improvement. Second, we show that the retrieval can be further improved by introducing an attention mechanism to the embedding learning model that reduces the effects of tempo variations in music. To conclude, we assess the scalability of our method and discuss potential measures to make it suitable for truly large-scale applications.