Multi-Format Contrastive Learning of Audio Representations

TL;DR

This paper demonstrates that multi-format contrastive learning, using raw audio and spectral representations, significantly improves audio representation quality and achieves state-of-the-art results on AudioSet and ESC-50 classification tasks.

Contribution

It introduces a novel multi-format contrastive learning approach for audio that leverages different representations of the same audio to enhance performance.

Findings

Significant performance gains over single-format methods.

Achieved state-of-the-art results on AudioSet and ESC-50.

Effective use of raw and spectral audio formats in contrastive learning.

Abstract

Recent advances suggest the advantage of multi-modal training in comparison with single-modal methods. In contrast to this view, in our work we find that similar gain can be obtained from training with different formats of a single modality. In particular, we investigate the use of the contrastive learning framework to learn audio representations by maximizing the agreement between the raw audio and its spectral representation. We find a significant gain using this multi-format strategy against the single-format counterparts. Moreover, on the downstream AudioSet and ESC-50 classification task, our audio-only approach achieves new state-of-the-art results with a mean average precision of 0.376 and an accuracy of 90.5%, respectively.