All-In-One Metrical And Functional Structure Analysis With Neighborhood Attentions on Demixed Audio

TL;DR

This paper presents a unified model that jointly performs beat, downbeat, and functional structure analysis of music, leveraging neighborhood attentions to improve accuracy across multiple hierarchical tasks.

Contribution

The paper introduces a versatile all-in-one model that captures hierarchical musical structures using neighborhood attentions, achieving state-of-the-art results on multiple tasks with fewer parameters.

Findings

State-of-the-art performance on all four tasks

Joint learning improves individual task accuracy

Model uses source-separated spectrograms and neighborhood attentions

Abstract

Music is characterized by complex hierarchical structures. Developing a comprehensive model to capture these structures has been a significant challenge in the field of Music Information Retrieval (MIR). Prior research has mainly focused on addressing individual tasks for specific hierarchical levels, rather than providing a unified approach. In this paper, we introduce a versatile, all-in-one model that jointly performs beat and downbeat tracking as well as functional structure segmentation and labeling. The model leverages source-separated spectrograms as inputs and employs dilated neighborhood attentions to capture temporal long-term dependencies, along with non-dilated attentions for local instrumental dependencies. Consequently, the proposed model achieves state-of-the-art performance in all four tasks on the Harmonix Set while maintaining a relatively lower number of parameters compared to recent state-of-the-art models. Furthermore, our ablation study demonstrates that the concurrent learning of beats, downbeats, and segments can lead to enhanced performance, with each task mutually benefiting from the others.