Echoes of Phonetics: Unveiling Relevant Acoustic Cues for ASR via Feature Attribution

TL;DR

This paper uses feature attribution to identify key acoustic cues in a modern ASR system, revealing how it processes different phonemes and highlighting differences between male and female speech.

Contribution

It applies a feature attribution technique to a state-of-the-art Conformer ASR model, providing detailed insights into the acoustic cues it relies on, which was not done in prior studies.

Findings

ASR relies on vowels' full time spans and first two formants, especially in male speech.

Spectral features of sibilant fricatives are more salient than non-sibilants.

The model emphasizes the release phase in plosives, focusing on burst characteristics.

Abstract

Despite significant advances in ASR, the specific acoustic cues models rely on remain unclear. Prior studies have examined such cues on a limited set of phonemes and outdated models. In this work, we apply a feature attribution technique to identify the relevant acoustic cues for a modern Conformer-based ASR system. By analyzing plosives, fricatives, and vowels, we assess how feature attributions align with their acoustic properties in the time and frequency domains, also essential for human speech perception. Our findings show that the ASR model relies on vowels' full time spans, particularly their first two formants, with greater saliency in male speech. It also better captures the spectral characteristics of sibilant fricatives than non-sibilants and prioritizes the release phase in plosives, especially burst characteristics. These insights enhance the interpretability of ASR models and highlight areas for future research to uncover potential gaps in model robustness.