Modeling Beats and Downbeats with a Time-Frequency Transformer

TL;DR

This paper introduces a novel Transformer-based model called SpecTNT for beat and downbeat tracking in music, leveraging spectral and temporal modeling to improve accuracy over existing methods.

Contribution

The paper proposes a new Spectral-Temporal Transformer architecture, SpecTNT, combined with TCN, to enhance beat and downbeat detection in music information retrieval.

Findings

SpecTNT outperforms TCN in downbeat tracking accuracy.

The combined SpecTNT and TCN model achieves state-of-the-art results.

The approach effectively models spectral and temporal features of music.

Abstract

Transformer is a successful deep neural network (DNN) architecture that has shown its versatility not only in natural language processing but also in music information retrieval (MIR). In this paper, we present a novel Transformer-based approach to tackle beat and downbeat tracking. This approach employs SpecTNT (Spectral-Temporal Transformer in Transformer), a variant of Transformer that models both spectral and temporal dimensions of a time-frequency input of music audio. A SpecTNT model uses a stack of blocks, where each consists of two levels of Transformer encoders. The lower-level (or spectral) encoder handles the spectral features and enables the model to pay attention to harmonic components of each frame. Since downbeats indicate bar boundaries and are often accompanied by harmonic changes, this step may help downbeat modeling. The upper-level (or temporal) encoder aggregates useful local spectral information to pay attention to beat/downbeat positions. We also propose an architecture that combines SpecTNT with a state-of-the-art model, Temporal Convolutional Networks (TCN), to further improve the performance. Extensive experiments demonstrate that our approach can significantly outperform TCN in downbeat tracking while maintaining comparable result in beat tracking.