Listenable Maps for Zero-Shot Audio Classifiers

TL;DR

This paper introduces LMAC-ZS, a novel decoder-based post-hoc interpretation method for zero-shot audio classifiers, enhancing transparency by providing faithful and meaningful explanations of model decisions.

Contribution

It presents the first decoder-based interpretation approach for zero-shot audio classifiers, utilizing a new loss function to improve faithfulness and explanation quality.

Findings

The method is faithful to the original model decisions.

It produces explanations that align well with different text prompts.

Extensive evaluation confirms effectiveness on CLAP model.

Abstract

Interpreting the decisions of deep learning models, including audio classifiers, is crucial for ensuring the transparency and trustworthiness of this technology. In this paper, we introduce LMAC-ZS (Listenable Maps for Audio Classifiers in the Zero-Shot context), which, to the best of our knowledge, is the first decoder-based post-hoc interpretation method for explaining the decisions of zero-shot audio classifiers. The proposed method utilizes a novel loss function that maximizes the faithfulness to the original similarity between a given text-and-audio pair. We provide an extensive evaluation using the Contrastive Language-Audio Pretraining (CLAP) model to showcase that our interpreter remains faithful to the decisions in a zero-shot classification context. Moreover, we qualitatively show that our method produces meaningful explanations that correlate well with different text prompts.