TONet: Tone-Octave Network for Singing Melody Extraction from Polyphonic Music

TL;DR

TONet is a novel neural network architecture that enhances singing melody extraction by explicitly modeling tone and octave information through a specialized input representation and fusion mechanism, outperforming existing methods.

Contribution

The paper introduces TONet, a plug-and-play model with a new input representation and fusion mechanism that significantly improves tone and octave perception in singing melody extraction.

Findings

Substantial improvements in octave and tone accuracy across datasets.

Effective use of Tone-CFP input representation for harmonic grouping.

Enhanced melody extraction performance with various backbone models.

Abstract

Singing melody extraction is an important problem in the field of music information retrieval. Existing methods typically rely on frequency-domain representations to estimate the sung frequencies. However, this design does not lead to human-level performance in the perception of melody information for both tone (pitch-class) and octave. In this paper, we propose TONet, a plug-and-play model that improves both tone and octave perceptions by leveraging a novel input representation and a novel network architecture. First, we present an improved input representation, the Tone-CFP, that explicitly groups harmonics via a rearrangement of frequency-bins. Second, we introduce an encoder-decoder architecture that is designed to obtain a salience feature map, a tone feature map, and an octave feature map. Third, we propose a tone-octave fusion mechanism to improve the final salience feature map. Experiments are done to verify the capability of TONet with various baseline backbone models. Our results show that tone-octave fusion with Tone-CFP can significantly improve the singing voice extraction performance across various datasets -- with substantial gains in octave and tone accuracy.