Contrastive and Transfer Learning for Effective Audio Fingerprinting through a Real-World Evaluation Protocol

TL;DR

This paper evaluates audio fingerprinting methods in real-world noisy conditions, introduces a new evaluation protocol, and demonstrates that a transformer-based model with transfer learning significantly outperforms CNN-based models.

Contribution

It presents a novel real-world evaluation protocol, highlights the importance of augmentation, and introduces a transformer model with transfer learning for robust audio fingerprinting.

Findings

Transformer model outperforms CNNs across noise levels and query durations.

Augmentation with filters improves CNN performance in noisy conditions.

Transfer learning from a relevant domain enhances robustness and accuracy.

Abstract

Recent advances in song identification leverage deep neural networks to learn compact audio fingerprints directly from raw waveforms. While these methods perform well under controlled conditions, their accuracy drops significantly in real-world scenarios where the audio is captured via mobile devices in noisy environments. In this paper, we introduce a novel evaluation protocol designed to better reflect such real-world conditions. We generate three recordings of the same audio, each with increasing levels of noise, captured using a mobile device's microphone. Our results reveal a substantial performance drop for two state-of-the-art CNN-based models under this protocol, compared to previously reported benchmarks. Additionally, we highlight the critical role of the augmentation pipeline during training with contrastive loss. By introduction low pass and high pass filters in the augmentation pipeline we significantly increase the performance of both systems in our proposed evaluation. Furthermore, we develop a transformer-based model with a tailored projection module and demonstrate that transferring knowledge from a semantically relevant domain yields a more robust solution. The transformer architecture outperforms CNN-based models across all noise levels, and query durations. In low noise conditions it achieves 47.99% for 1-sec queries, and 97% for 10-sec queries in finding the correct song, surpassing by 14%, and by 18.5% the second-best performing model, respectively, Under heavy noise levels, we achieve a detection rate 56.5% for 15-second query duration. All experiments are conducted on public large-scale dataset of over 100K songs, with queries matched against a database of 56 million vectors.