Similarity measures for vocal-based drum sample retrieval using deep convolutional auto-encoders

TL;DR

This study evaluates how deep convolutional auto-encoders can learn features that effectively predict perceptual similarity between vocal imitations and drum sounds, outperforming traditional features.

Contribution

It demonstrates that features learned by convolutional auto-encoders better predict perceptual similarity in vocal-based drum sample retrieval than baseline methods.

Findings

CAEs outperform spectrograms, MFCCs, and temporal features in predicting similarity.

Preservation of temporal information is more crucial than spectral resolution.

Learned features' effectiveness depends on the size and shape of the encoded layer.

Abstract

The expressive nature of the voice provides a powerful medium for communicating sonic ideas, motivating recent research on methods for query by vocalisation. Meanwhile, deep learning methods have demonstrated state-of-the-art results for matching vocal imitations to imitated sounds, yet little is known about how well learned features represent the perceptual similarity between vocalisations and queried sounds. In this paper, we address this question using similarity ratings between vocal imitations and imitated drum sounds. We use a linear mixed effect regression model to show how features learned by convolutional auto-encoders (CAEs) perform as predictors for perceptual similarity between sounds. Our experiments show that CAEs outperform three baseline feature sets (spectrogram-based representations, MFCCs, and temporal features) at predicting the subjective similarity ratings. We also investigate how the size and shape of the encoded layer effects the predictive power of the learned features. The results show that preservation of temporal information is more important than spectral resolution for this application.