Lombard Effect for Bilingual Speakers in Cantonese and English: importance of spectro-temporal features

TL;DR

This study investigates how spectro-temporal features influence speech recognition in Cantonese and English, highlighting their importance in modeling Lombard speech effects and noise conditions using an ASR system.

Contribution

It demonstrates that spectro-temporal features are essential for predicting speech recognition thresholds and Lombard speech effects across tonal and non-tonal languages.

Findings

Spectro-temporal features improve prediction of speech recognition thresholds.

Spectral features can predict Lombard speech effects.

Features are crucial under various noise conditions.

Abstract

For a better understanding of the mechanisms underlying speech perception and the contribution of different signal features, computational models of speech recognition have a long tradition in hearing research. Due to the diverse range of situations in which speech needs to be recognized, these models need to be generalizable across many acoustic conditions, speakers, and languages. This contribution examines the importance of different features for speech recognition predictions of plain and Lombard speech for English in comparison to Cantonese in stationary and modulated noise. While Cantonese is a tonal language that encodes information in spectro-temporal features, the Lombard effect is known to be associated with spectral changes in the speech signal. These contrasting properties of tonal languages and the Lombard effect form an interesting basis for the assessment of speech recognition models. Here, an automatic speech recognition-based ASR model using spectral or spectro-temporal features is evaluated with empirical data. The results indicate that spectro-temporal features are crucial in order to predict the speaker-specific speech recognition threshold SRT$_{50}$ in both Cantonese and English as well as to account for the improvement of speech recognition in modulated noise, while effects due to Lombard speech can already be predicted by spectral features.