Uncovering the role of semantic and acoustic cues in normal and dichotic listening

TL;DR

This study investigates how acoustic and semantic cues contribute to speech comprehension in normal and dichotic listening, using EEG data and a novel deep learning model to quantify their roles.

Contribution

The paper introduces a multimodal deep learning model and a match-mismatch task to quantify acoustic and semantic cue roles in speech perception under different listening conditions.

Findings

Semantic cues improve MM classification in dichotic listening.

Speech perception is fragmented at word boundaries.

The proposed model outperforms previous MM models.

Abstract

Speech comprehension is an involuntary task for the healthy human brain, yet the understanding of the mechanisms underlying this brain functionality remains obscure. In this paper, we aim to quantify the role of acoustic and semantic information streams in complex listening conditions. We propose a paradigm to understand the encoding of the speech cues in electroencephalogram (EEG) data, by designing a match-mismatch (MM) classification task. The MM task involves identifying whether the stimulus (speech) and response (EEG) correspond to each other. We build a multimodal deep-learning based sequence model STEM, which is input with acoustic stimulus (speech envelope), semantic stimulus (textual representations of speech), and the neural response (EEG data). We perform extensive experiments on two separate conditions, i) natural passive listening and, ii) a dichotic listening requiring auditory attention. Using the MM task as the analysis framework, we observe that - a) speech perception is fragmented based on word boundaries, b) acoustic and semantic cues offer similar levels of MM task performance in natural listening conditions, and c) semantic cues offer significantly improved MM classification over acoustic cues in dichotic listening task. The comparison of the STEM with previously proposed MM models shows significant performance improvements for the proposed approach. The analysis and understanding from this study allows the quantification of the roles played by acoustic and semantic cues in diverse listening tasks and in providing further evidences of right-ear advantage in dichotic listening.