Directional Source Separation for Robust Speech Recognition on Smart Glasses

TL;DR

This paper explores directional source separation techniques, including neural beamforming, to enhance speech recognition accuracy on smart glasses in noisy environments, demonstrating improved ASR performance for the user.

Contribution

It introduces neural beamforming for source separation and joint training with ASR, advancing noise robustness in smart glasses speech recognition systems.

Findings

Directional source separation improves ASR for the wearer.

Neural beamforming effectively learns directional characteristics.

Joint training yields the best overall ASR performance.

Abstract

Modern smart glasses leverage advanced audio sensing and machine learning technologies to offer real-time transcribing and captioning services, considerably enriching human experiences in daily communications. However, such systems frequently encounter challenges related to environmental noises, resulting in degradation to speech recognition and speaker change detection. To improve voice quality, this work investigates directional source separation using the multi-microphone array. We first explore multiple beamformers to assist source separation modeling by strengthening the directional properties of speech signals. In addition to relying on predetermined beamformers, we investigate neural beamforming in multi-channel source separation, demonstrating that automatic learning directional characteristics effectively improves separation quality. We further compare the ASR performance leveraging separated outputs to noisy inputs. Our results show that directional source separation benefits ASR for the wearer but not for the conversation partner. Lastly, we perform the joint training of the directional source separation and ASR model, achieving the best overall ASR performance.