Exploiting Time-Frequency Conformers for Music Audio Enhancement

TL;DR

This paper introduces a novel music enhancement system based on the Conformer architecture, leveraging attention mechanisms to improve degraded music recordings, achieving state-of-the-art results especially in multi-track scenarios.

Contribution

The paper presents the first application of Conformer-based models to music enhancement, exploring attention mechanisms for improved performance on single-stem and multi-track music recordings.

Findings

Achieves state-of-the-art performance on single-stem music enhancement

Successfully extends to multi-track music enhancement tasks

Demonstrates the effectiveness of Conformer architecture in music audio processing

Abstract

With the proliferation of video platforms on the internet, recording musical performances by mobile devices has become commonplace. However, these recordings often suffer from degradation such as noise and reverberation, which negatively impact the listening experience. Consequently, the necessity for music audio enhancement (referred to as music enhancement from this point onward), involving the transformation of degraded audio recordings into pristine high-quality music, has surged to augment the auditory experience. To address this issue, we propose a music enhancement system based on the Conformer architecture that has demonstrated outstanding performance in speech enhancement tasks. Our approach explores the attention mechanisms of the Conformer and examines their performance to discover the best approach for the music enhancement task. Our experimental results show that our proposed model achieves state-of-the-art performance on single-stem music enhancement. Furthermore, our system can perform general music enhancement with multi-track mixtures, which has not been examined in previous work.