Learning spectro-temporal representations of complex sounds with parameterized neural networks

TL;DR

This paper introduces a fully interpretable, parameterized neural network layer that learns spectro-temporal modulations for auditory tasks, matching or surpassing existing models and aligning with biological auditory features.

Contribution

The authors propose a novel, interpretable neural network layer based on Gabor kernels that effectively models spectro-temporal features across diverse auditory tasks.

Findings

Learnable STRFs perform on par with state-of-the-art models.

Filters focus on low temporal and spectral modulations.

Learned features resemble those in human auditory cortex.

Abstract

Deep Learning models have become potential candidates for auditory neuroscience research, thanks to their recent successes on a variety of auditory tasks. Yet, these models often lack interpretability to fully understand the exact computations that have been performed. Here, we proposed a parametrized neural network layer, that computes specific spectro-temporal modulations based on Gabor kernels (Learnable STRFs) and that is fully interpretable. We evaluated predictive capabilities of this layer on Speech Activity Detection, Speaker Verification, Urban Sound Classification and Zebra Finch Call Type Classification. We found out that models based on Learnable STRFs are on par for all tasks with different toplines, and obtain the best performance for Speech Activity Detection. As this layer is fully interpretable, we used quantitative measures to describe the distribution of the learned spectro-temporal modulations. The filters adapted to each task and focused mostly on low temporal and spectral modulations. The analyses show that the filters learned on human speech have similar spectro-temporal parameters as the ones measured directly in the human auditory cortex. Finally, we observed that the tasks organized in a meaningful way: the human vocalizations tasks closer to each other and bird vocalizations far away from human vocalizations and urban sounds tasks.