Non-Intrusive Speech Intelligibility Prediction for Hearing Aids using Whisper and Metadata

TL;DR

This paper introduces MBI-Net+ with Whisper embeddings, speech metadata, and HASPI integration, significantly improving speech intelligibility prediction accuracy for hearing aids over previous models.

Contribution

The paper presents three novel methods and an enhanced model, MBI-Net+, that improve speech intelligibility prediction accuracy by leveraging Whisper embeddings, metadata, and HASPI metrics.

Findings

MBI-Net+ outperforms baseline systems on Clarity Prediction Challenge 2023 dataset.

Incorporating Whisper embeddings improves acoustic feature representation.

Using HASPI as a supplementary metric enhances prediction performance.

Abstract

Automated speech intelligibility assessment is pivotal for hearing aid (HA) development. In this paper, we present three novel methods to improve intelligibility prediction accuracy and introduce MBI-Net+, an enhanced version of MBI-Net, the top-performing system in the 1st Clarity Prediction Challenge. MBI-Net+ leverages Whisper's embeddings to create cross-domain acoustic features and includes metadata from speech signals by using a classifier that distinguishes different enhancement methods. Furthermore, MBI-Net+ integrates the hearing-aid speech perception index (HASPI) as a supplementary metric into the objective function to further boost prediction performance. Experimental results demonstrate that MBI-Net+ surpasses several intrusive baseline systems and MBI-Net on the Clarity Prediction Challenge 2023 dataset, validating the effectiveness of incorporating Whisper embeddings, speech metadata, and related complementary metrics to improve prediction performance for HA.