Listen to Interpret: Post-hoc Interpretability for Audio Networks with NMF

TL;DR

This paper introduces a novel post-hoc interpretability method for audio neural networks using non-negative matrix factorization, enabling intuitive, listenable explanations of model decisions based on high-level audio components.

Contribution

It proposes a new NMF-based interpreter that produces listenable audio explanations from hidden layer representations, enhancing interpretability of audio processing networks.

Findings

Effective in generating listenable audio explanations

Applicable to multi-label classification benchmarks

Improves understanding of network decision-making

Abstract

This paper tackles post-hoc interpretability for audio processing networks. Our goal is to interpret decisions of a network in terms of high-level audio objects that are also listenable for the end-user. To this end, we propose a novel interpreter design that incorporates non-negative matrix factorization (NMF). In particular, a carefully regularized interpreter module is trained to take hidden layer representations of the targeted network as input and produce time activations of pre-learnt NMF components as intermediate outputs. Our methodology allows us to generate intuitive audio-based interpretations that explicitly enhance parts of the input signal most relevant for a network's decision. We demonstrate our method's applicability on popular benchmarks, including a real-world multi-label classification task.