Spectrotemporal Modulation: Efficient and Interpretable Feature Representation for Classifying Speech, Music, and Environmental Sounds

TL;DR

This paper introduces spectrotemporal modulation features as an efficient, interpretable alternative to deep neural network representations for classifying speech, music, and environmental sounds, matching the performance of pretrained models.

Contribution

It presents a novel STM-based feature extraction method inspired by human auditory processing, achieving comparable classification results without pretraining.

Findings

STM features match pretrained DNN performance

STM offers computational efficiency and interpretability

Applicable across speech, music, and environmental sounds

Abstract

Audio DNNs have demonstrated impressive performance on various machine listening tasks; however, most of their representations are computationally costly and uninterpretable, leaving room for optimization. Here, we propose a novel approach centered on spectrotemporal modulation (STM) features, a signal processing method that mimics the neurophysiological representation in the human auditory cortex. The classification performance of our STM-based model, without any pretraining, is comparable to that of pretrained audio DNNs across diverse naturalistic speech, music, and environmental sounds, which are essential categories for both human cognition and machine perception. These results show that STM is an efficient and interpretable feature representation for audio classification, advancing the development of machine listening and unlocking exciting new possibilities for basic understanding of speech and auditory sciences, as well as developing audio BCI and cognitive computing.