Modeling Multi-Level Hearing Loss for Speech Intelligibility Prediction

TL;DR

This paper introduces a novel speech intelligibility prediction method that models hearing loss effects by simulating auditory degradations and analyzing spectro-temporal modulations, outperforming existing indices especially for hearing-impaired listeners.

Contribution

It presents a new auditory-inspired feature extraction and a Transformer-based regression model that explicitly accounts for hearing loss severity, improving speech intelligibility prediction accuracy.

Findings

Outperforms HASPI v2 in predicting speech intelligibility for hearing-impaired groups.

Reduces prediction error by 16.5% for mild hearing loss and 6.1% for moderate-to-severe loss.

Highlights the importance of modeling frequency and temporal resolution deficits.

Abstract

The diverse perceptual consequences of hearing loss severely impede speech communication, but standard clinical audiometry, which is focused on threshold-based frequency sensitivity, does not adequately capture deficits in frequency and temporal resolution. To address this limitation, we propose a speech intelligibility prediction method that explicitly simulates auditory degradations according to hearing loss severity by broadening cochlear filters and applying low-pass modulation filtering to temporal envelopes. Speech signals are subsequently analyzed using the spectro-temporal modulation (STM) representations, which reflect how auditory resolution loss alters the underlying modulation structure. In addition, normalized cross-correlation (NCC) matrices quantify the similarity between the STM representations of clean speech and speech in noise. These auditory-informed features are utilized to train a Vision Transformer-based regression model that integrates the STM maps and NCC embeddings to estimate speech intelligibility scores. Evaluations on the Clarity Prediction Challenge corpus show that the proposed method outperforms the Hearing-Aid Speech Perception Index v2 (HASPI v2) in both mild and moderate-to-severe hearing loss groups, with a relative root mean squared error reduction of 16.5% for the mild group and a 6.1% reduction for the moderate-to-severe group. These results highlight the importance of explicitly modeling listener-specific frequency and temporal resolution degradations to improve speech intelligibility prediction and provide interpretability in auditory distortions.