Investigating an Overfitting and Degeneration Phenomenon in Self-Supervised Multi-Pitch Estimation

TL;DR

This paper explores the use of self-supervised learning in multi-pitch estimation, revealing a phenomenon where models overfit supervised data while degenerating on self-supervised data, and offers insights into this issue.

Contribution

It introduces self-supervised objectives into the MPE paradigm and investigates the overfitting and degeneration phenomena encountered during joint training.

Findings

Self-supervised objectives improve performance under closed conditions.

Overfitting to supervised data causes degeneration on self-supervised data.

Insights into the overfitting and degeneration phenomena are provided.

Abstract

Multi-Pitch Estimation (MPE) continues to be a sought after capability of Music Information Retrieval (MIR) systems, and is critical for many applications and downstream tasks involving pitch, including music transcription. However, existing methods are largely based on supervised learning, and there are significant challenges in collecting annotated data for the task. Recently, self-supervised techniques exploiting intrinsic properties of pitch and harmonic signals have shown promise for both monophonic and polyphonic pitch estimation, but these still remain inferior to supervised methods. In this work, we extend the classic supervised MPE paradigm by incorporating several self-supervised objectives based on pitch-invariant and pitch-equivariant properties. This joint training results in a substantial improvement under closed training conditions, which naturally suggests that applying the same objectives to a broader collection of data will yield further improvements. However, in doing so we uncover a phenomenon whereby our model simultaneously overfits to the supervised data while degenerating on data used for self-supervision only. We demonstrate and investigate this and offer our insights on the underlying problem.