How Does a Deep Neural Network Look at Lexical Stress in English Words?

TL;DR

This study investigates how convolutional neural networks interpret lexical stress in English words, revealing that they primarily rely on spectral features of stressed vowels, thus providing insights into neural decision-making in speech processing.

Contribution

The paper introduces an interpretability analysis of CNNs trained on natural speech data for lexical stress prediction, highlighting the spectral cues used by deep models, especially formant information.

Findings

CNNs achieved up to 92% accuracy in stress prediction.

Predictions are mainly influenced by spectral features of stressed vowels.

Deep learning captures distributed cues to stress from natural speech.

Abstract

Despite their success in speech processing, neural networks often operate as black boxes, prompting the question: what informs their decisions, and how can we interpret them? This work examines this issue in the context of lexical stress. A dataset of English disyllabic words was automatically constructed from read and spontaneous speech. Several Convolutional Neural Network (CNN) architectures were trained to predict stress position from a spectrographic representation of disyllabic words lacking minimal stress pairs (e.g., initial stress WAllet, final stress exTEND), achieving up to 92% accuracy on held-out test data. Layerwise Relevance Propagation (LRP), a technique for neural network interpretability analysis, revealed that predictions for held-out minimal pairs (PROtest vs. proTEST ) were most strongly influenced by information in stressed versus unstressed syllables, particularly the spectral properties of stressed vowels. However, the classifiers also attended to information throughout the word. A feature-specific relevance analysis is proposed, and its results suggest that our best-performing classifier is strongly influenced by the stressed vowel's first and second formants, with some evidence that its pitch and third formant also contribute. These results reveal deep learning's ability to acquire distributed cues to stress from naturally occurring data, extending traditional phonetic work based around highly controlled stimuli.