Hybrid Transformers for Music Source Separation

TL;DR

This paper introduces Hybrid Transformer Demucs, a novel model combining Transformers with a bi-U-Net architecture for music source separation, achieving state-of-the-art results with additional training data.

Contribution

It presents a hybrid temporal/spectral Transformer integrated into a bi-U-Net architecture for improved music source separation performance.

Findings

Outperforms Hybrid Demucs with extra training data by 0.45 dB SDR.

Achieves 9.20 dB SDR on MUSDB with extended training.

Utilizes sparse attention kernels and per source fine-tuning for optimal results.

Abstract

A natural question arising in Music Source Separation (MSS) is whether long range contextual information is useful, or whether local acoustic features are sufficient. In other fields, attention based Transformers have shown their ability to integrate information over long sequences. In this work, we introduce Hybrid Transformer Demucs (HT Demucs), an hybrid temporal/spectral bi-U-Net based on Hybrid Demucs, where the innermost layers are replaced by a cross-domain Transformer Encoder, using self-attention within one domain, and cross-attention across domains. While it performs poorly when trained only on MUSDB, we show that it outperforms Hybrid Demucs (trained on the same data) by 0.45 dB of SDR when using 800 extra training songs. Using sparse attention kernels to extend its receptive field, and per source fine-tuning, we achieve state-of-the-art results on MUSDB with extra training data, with 9.20 dB of SDR.