A Study of Joint Effect on Denoising Techniques and Visual Cues to Improve Speech Intelligibility in Cochlear Implant Simulation

TL;DR

This study investigates how combining advanced neural network-based speech enhancement with visual cues can significantly improve speech intelligibility in cochlear implant simulations, especially in noisy environments.

Contribution

It demonstrates the effectiveness of FCN-based denoising and audiovisual integration in enhancing vocoded speech perception, proposing a combined approach for cochlear implant processing.

Findings

FCN-based denoising improves speech clarity in CI simulation.

Audiovisual cues enhance speech comprehension over audio-only.

Combined methods show potential for CI device integration.

Abstract

Speech perception is key to verbal communication. For people with hearing loss, the capability to recognize speech is restricted, particularly in a noisy environment or the situations without visual cues, such as lip-reading unavailable via phone call. This study aimed to understand the improvement of vocoded speech intelligibility in cochlear implant (CI) simulation through two potential methods: Speech Enhancement (SE) and Audiovisual Integration. A fully convolutional neural network (FCN) using an intelligibility-oriented objective function was recently proposed and proven to effectively facilitate the speech intelligibility as an advanced denoising SE approach. Furthermore, audiovisual integration is reported to supply better speech comprehension compared to audio-only information. An experiment was designed to test speech intelligibility using tone-vocoded speech in CI simulation with a group of normal-hearing listeners. Experimental results confirmed the effectiveness of the FCN-based denoising SE and audiovisual integration on vocoded speech. Also, it positively recommended that these two methods could become a blended feature in a CI processor to improve the speech intelligibility for CI users under noisy conditions.