Unveiling the Best Practices for Applying Speech Foundation Models to Speech Intelligibility Prediction for Hearing-Impaired People

TL;DR

This study systematically investigates how to optimize speech foundation models for predicting speech intelligibility in hearing-impaired individuals, revealing key design choices that enhance performance.

Contribution

It provides the first comprehensive analysis of design factors affecting SFM-based speech intelligibility prediction for hearing-impaired people, including layer selection, temporal modeling, and ensembling.

Findings

Single encoder layer selection outperforms all-layers approach.

Temporal modeling significantly improves prediction accuracy.

Ensembling multiple SFMs enhances performance, especially with stronger models.

Abstract

Speech foundation models (SFMs) have demonstrated strong performance across a variety of downstream tasks, including speech intelligibility prediction for hearing-impaired people (SIP-HI). However, optimizing SFMs for SIP-HI has been insufficiently explored. In this paper, we conduct a comprehensive study to identify key design factors affecting SIP-HI performance with 5 SFMs, focusing on encoder layer selection, prediction head architecture, and ensemble configurations. Our findings show that, contrary to traditional use-all-layers methods, selecting a single encoder layer yields better results. Additionally, temporal modeling is crucial for effective prediction heads. We also demonstrate that ensembling multiple SFMs improves performance, with stronger individual models providing greater benefit. Finally, we explore the relationship between key SFM attributes and their impact on SIP-HI performance. Our study offers practical insights into effectively adapting SFMs for speech intelligibility prediction for hearing-impaired populations.