Contrastive Learning from Synthetic Audio Doppelg\"angers

TL;DR

This paper introduces a contrastive learning method using synthetic audio generated by perturbing sound synthesizer parameters, which improves audio representation quality and reduces data requirements.

Contribution

It presents a novel approach of using synthetic audio pairs for contrastive learning, outperforming real data methods and requiring minimal hyperparameters.

Findings

Synthetic audio pairs enhance contrastive learning effectiveness.

Method outperforms real data-based approaches on standard tasks.

Approach is lightweight with no data storage needs.

Abstract

Learning robust audio representations currently demands extensive datasets of real-world sound recordings. By applying artificial transformations to these recordings, models can learn to recognize similarities despite subtle variations through techniques like contrastive learning. However, these transformations are only approximations of the true diversity found in real-world sounds, which are generated by complex interactions of physical processes, from vocal cord vibrations to the resonance of musical instruments. We propose a solution to both the data scale and transformation limitations, leveraging synthetic audio. By randomly perturbing the parameters of a sound synthesizer, we generate audio doppelg\"angers-synthetic positive pairs with causally manipulated variations in timbre, pitch, and temporal envelopes. These variations, difficult to achieve through augmentations of existing audio, provide a rich source of contrastive information. Despite the shift to randomly generated synthetic data, our method produces strong representations, outperforming real data on several standard audio classification tasks. Notably, our approach is lightweight, requires no data storage, and has only a single hyperparameter, which we extensively analyze. We offer this method as a complement to existing strategies for contrastive learning in audio, using synthesized sounds to reduce the data burden on practitioners.