Explaining Speech Classification Models via Word-Level Audio Segments and Paralinguistic Features

TL;DR

This paper introduces a novel method for explaining speech classification models by analyzing word-level audio segments and paralinguistic features, making model decisions more interpretable for users.

Contribution

It presents a new input perturbation approach for generating interpretable explanations at both word and paralinguistic levels in speech models.

Findings

Explanations are faithful to model inner workings.

Explanations are plausible and understandable to humans.

Method validated on English and Italian speech tasks.

Abstract

Recent advances in eXplainable AI (XAI) have provided new insights into how models for vision, language, and tabular data operate. However, few approaches exist for understanding speech models. Existing work focuses on a few spoken language understanding (SLU) tasks, and explanations are difficult to interpret for most users. We introduce a new approach to explain speech classification models. We generate easy-to-interpret explanations via input perturbation on two information levels. 1) Word-level explanations reveal how each word-related audio segment impacts the outcome. 2) Paralinguistic features (e.g., prosody and background noise) answer the counterfactual: ``What would the model prediction be if we edited the audio signal in this way?'' We validate our approach by explaining two state-of-the-art SLU models on two speech classification tasks in English and Italian. Our findings demonstrate that the explanations are faithful to the model's inner workings and plausible to humans. Our method and findings pave the way for future research on interpreting speech models.