Using Speech Foundational Models in Loss Functions for Hearing Aid Speech Enhancement

TL;DR

This paper introduces a novel loss function for speech enhancement in hearing aids that leverages self-supervised speech representations, leading to improved intelligibility metrics without increasing inference complexity.

Contribution

It demonstrates that using self-supervised speech representation distances as a loss function enhances speech intelligibility in hearing aid applications.

Findings

Improved HASPI, STOI, PESQ, and SI-SNR scores with the new loss function.

Representation-based loss correlates more strongly with human intelligibility.

Model complexity remains low during inference, suitable for hearing aids.

Abstract

Machine learning techniques are an active area of research for speech enhancement for hearing aids, with one particular focus on improving the intelligibility of a noisy speech signal. Recent work has shown that feature encodings from self-supervised speech representation models can effectively capture speech intelligibility. In this work, it is shown that the distance between self-supervised speech representations of clean and noisy speech correlates more strongly with human intelligibility ratings than other signal-based metrics. Experiments show that training a speech enhancement model using this distance as part of a loss function improves the performance over using an SNR-based loss function, demonstrated by an increase in HASPI, STOI, PESQ and SI-SNR scores. This method takes inference of a high parameter count model only at training time, meaning the speech enhancement model can remain smaller, as is required for hearing aids.