PESTO: Pitch Estimation with Self-supervised Transposition-equivariant Objective

TL;DR

This paper introduces PESTO, a lightweight self-supervised model for pitch estimation that leverages transposition-equivariance, enabling accurate, real-time pitch detection on low-resource devices with minimal labeled data.

Contribution

The paper proposes a novel transposition-equivariant SSL framework with a transposition-preserving architecture for pitch estimation, reducing reliance on labeled data and improving cross-task generalization.

Findings

Outperforms existing self-supervised baselines in pitch estimation accuracy.

Generalizes effectively across different musical tasks and datasets.

Operates efficiently on low-resource, real-time systems.

Abstract

In this paper, we address the problem of pitch estimation using Self Supervised Learning (SSL). The SSL paradigm we use is equivariance to pitch transposition, which enables our model to accurately perform pitch estimation on monophonic audio after being trained only on a small unlabeled dataset. We use a lightweight ($<$ 30k parameters) Siamese neural network that takes as inputs two different pitch-shifted versions of the same audio represented by its Constant-Q Transform. To prevent the model from collapsing in an encoder-only setting, we propose a novel class-based transposition-equivariant objective which captures pitch information. Furthermore, we design the architecture of our network to be transposition-preserving by introducing learnable Toeplitz matrices. We evaluate our model for the two tasks of singing voice and musical instrument pitch estimation and show that our model is able to generalize across tasks and datasets while being lightweight, hence remaining compatible with low-resource devices and suitable for real-time applications. In particular, our results surpass self-supervised baselines and narrow the performance gap between self-supervised and supervised methods for pitch estimation.