Non-Intrusive Binaural Speech Intelligibility Prediction from Discrete Latent Representations

TL;DR

This paper introduces a non-intrusive binaural speech intelligibility prediction method using vector quantization and contrastive predictive coding, which outperforms existing benchmarks without relying on clean speech signals.

Contribution

It proposes a novel non-intrusive SI measure based on VQ-CPC features that effectively captures binaural information without requiring auditory system models.

Findings

VQ-CPC features outperform benchmarks in correlation and MSE.

Method generalizes well to different noise types.

Features effectively model speech intelligibility.

Abstract

Non-intrusive speech intelligibility (SI) prediction from binaural signals is useful in many applications. However, most existing signal-based measures are designed to be applied to single-channel signals. Measures specifically designed to take into account the binaural properties of the signal are often intrusive - characterised by requiring access to a clean speech signal - and typically rely on combining both channels into a single-channel signal before making predictions. This paper proposes a non-intrusive SI measure that computes features from a binaural input signal using a combination of vector quantization (VQ) and contrastive predictive coding (CPC) methods. VQ-CPC feature extraction does not rely on any model of the auditory system and is instead trained to maximise the mutual information between the input signal and output features. The computed VQ-CPC features are input to a predicting function parameterized by a neural network. Two predicting functions are considered in this paper. Both feature extractor and predicting functions are trained on simulated binaural signals with isotropic noise. They are tested on simulated signals with isotropic and real noise. For all signals, the ground truth scores are the (intrusive) deterministic binaural STOI. Results are presented in terms of correlations and MSE and demonstrate that VQ-CPC features are able to capture information relevant to modelling SI and outperform all the considered benchmarks - even when evaluating on data comprising of different noise field types.