S3T: Self-Supervised Pre-training with Swin Transformer for Music Classification

TL;DR

S3T introduces a novel self-supervised pre-training approach using Swin Transformer and MoCo for music classification, significantly improving accuracy and label efficiency over previous methods.

Contribution

It is the first to combine Swin Transformer with self-supervised learning for music classification, incorporating a new data augmentation pipeline and pre-processors.

Findings

Outperforms previous self-supervised method CLMR by 12.5% top-1 accuracy

Surpasses state-of-the-art supervised methods on music tasks

Achieves high performance with only 10% labeled data

Abstract

In this paper, we propose S3T, a self-supervised pre-training method with Swin Transformer for music classification, aiming to learn meaningful music representations from massive easily accessible unlabeled music data. S3T introduces a momentum-based paradigm, MoCo, with Swin Transformer as its feature extractor to music time-frequency domain. For better music representations learning, S3T contributes a music data augmentation pipeline and two specially designed pre-processors. To our knowledge, S3T is the first method combining the Swin Transformer with a self-supervised learning method for music classification. We evaluate S3T on music genre classification and music tagging tasks with linear classifiers trained on learned representations. Experimental results show that S3T outperforms the previous self-supervised method (CLMR) by 12.5 percents top-1 accuracy and 4.8 percents PR-AUC on two tasks respectively, and also surpasses the task-specific state-of-the-art supervised methods. Besides, S3T shows advances in label efficiency using only 10% labeled data exceeding CLMR on both tasks with 100% labeled data.