Direction-Aware Joint Adaptation of Neural Speech Enhancement and Recognition in Real Multiparty Conversational Environments

TL;DR

This paper introduces a direction-aware joint adaptation method for neural speech enhancement and recognition in real multiparty environments, improving recognition accuracy for AR headsets amid dynamic conditions.

Contribution

It proposes a semi-supervised, joint adaptation technique that updates both speech mask estimation and ASR models in real-time, addressing training-test mismatch and head movement challenges.

Findings

Significant ASR performance improvements demonstrated

Effective adaptation in real multiparty scenarios

Enhanced robustness to head movements and environmental noise

Abstract

This paper describes noisy speech recognition for an augmented reality headset that helps verbal communication within real multiparty conversational environments. A major approach that has actively been studied in simulated environments is to sequentially perform speech enhancement and automatic speech recognition (ASR) based on deep neural networks (DNNs) trained in a supervised manner. In our task, however, such a pretrained system fails to work due to the mismatch between the training and test conditions and the head movements of the user. To enhance only the utterances of a target speaker, we use beamforming based on a DNN-based speech mask estimator that can adaptively extract the speech components corresponding to a head-relative particular direction. We propose a semi-supervised adaptation method that jointly updates the mask estimator and the ASR model at run-time using clean speech signals with ground-truth transcriptions and noisy speech signals with highly-confident estimated transcriptions. Comparative experiments using the state-of-the-art distant speech recognition system show that the proposed method significantly improves the ASR performance.