Pretrained Conformers for Audio Fingerprinting and Retrieval

TL;DR

This paper introduces a self-supervised conformer-based encoder that generates robust audio embeddings for retrieval, achieving state-of-the-art results with minimal audio segments and high resilience to distortions.

Contribution

It presents a novel self-supervised contrastive learning framework for conformers that produces highly effective audio embeddings for retrieval tasks, even with limited audio data.

Findings

State-of-the-art audio retrieval performance with 3-second segments.

Models are highly resistant to temporal misalignments and distortions.

Code and models are publicly available for reproducibility.

Abstract

Conformers have shown great results in speech processing due to their ability to capture both local and global interactions. In this work, we utilize a self-supervised contrastive learning framework to train conformer-based encoders that are capable of generating unique embeddings for small segments of audio, generalizing well to previously unseen data. We achieve state-of-the-art results for audio retrieval tasks while using only 3 seconds of audio to generate embeddings. Our models are almost completely immune to temporal misalignments and achieve state-of-the-art results in cases of other audio distortions such as noise, reverb or extreme temporal stretching. Code and models are made publicly available and the results are easy to reproduce as we train and test using popular and freely available datasets of different sizes.