Show Me the Instruments: Musical Instrument Retrieval from Mixture Audio

TL;DR

This paper introduces a neural network-based method for retrieving specific musical instruments from mixture audio using reference samples, facilitating more efficient music production and instrument selection.

Contribution

It proposes a novel dual-encoder neural network architecture for instrument retrieval from mixture audio and introduces a new dataset called Nlakh for training and evaluation.

Findings

Single-Instrument Encoder effectively maps unseen instruments to embedding space.

Multi-Instrument Encoder successfully extracts multiple instrument embeddings from mixtures.

The method achieves accurate retrieval of desired instruments from complex audio mixtures.

Abstract

As digital music production has become mainstream, the selection of appropriate virtual instruments plays a crucial role in determining the quality of music. To search the musical instrument samples or virtual instruments that make one's desired sound, music producers use their ears to listen and compare each instrument sample in their collection, which is time-consuming and inefficient. In this paper, we call this task as Musical Instrument Retrieval and propose a method for retrieving desired musical instruments using reference music mixture as a query. The proposed model consists of the Single-Instrument Encoder and the Multi-Instrument Encoder, both based on convolutional neural networks. The Single-Instrument Encoder is trained to classify the instruments used in single-track audio, and we take its penultimate layer's activation as the instrument embedding. The Multi-Instrument Encoder is trained to estimate multiple instrument embeddings using the instrument embeddings computed by the Single-Instrument Encoder as a set of target embeddings. For more generalized training and realistic evaluation, we also propose a new dataset called Nlakh. Experimental results showed that the Single-Instrument Encoder was able to learn the mapping from the audio signal of unseen instruments to the instrument embedding space and the Multi-Instrument Encoder was able to extract multiple embeddings from the mixture of music and retrieve the desired instruments successfully. The code used for the experiment and audio samples are available at: https://github.com/minju0821/musical_instrument_retrieval