Musical Metamerism with Time--Frequency Scattering

TL;DR

This paper introduces a method to generate musical metamers using joint time-frequency scattering, enabling auditory similarity perception despite differences in waveforms, without manual preprocessing.

Contribution

It presents a novel approach leveraging joint time-frequency scattering for musical metamerism, avoiding manual audio preprocessing steps.

Findings

Method successfully generates musical metamers.

No manual transcription or source separation needed.

Connects JTFS with existing auditory models.

Abstract

The concept of metamerism originates from colorimetry, where it describes a sensation of visual similarity between two colored lights despite significant differences in spectral content. Likewise, we propose to call ``musical metamerism'' the sensation of auditory similarity which is elicited by two music fragments which differ in terms of underlying waveforms. In this technical report, we describe a method to generate musical metamers from any audio recording. Our method is based on joint time--frequency scattering in Kymatio, an open-source software in Python which enables GPU computing and automatic differentiation. The advantage of our method is that it does not require any manual preprocessing, such as transcription, beat tracking, or source separation. We provide a mathematical description of JTFS as well as some excerpts from the Kymatio source code. Lastly, we review the prior work on JTFS and draw connections with closely related algorithms, such as spectrotemporal receptive fields (STRF), modulation power spectra (MPS), and Gabor filterbank (GBFB).