Ensemble of Jointly Trained Deep Neural Network-Based Acoustic Models for Reverberant Speech Recognition

TL;DR

This paper introduces an ensemble of jointly trained deep neural networks tailored for reverberant speech recognition, improving accuracy across diverse reverberation conditions by combining models trained for different RT60s and dereverberation.

Contribution

It proposes a novel ensemble approach with joint training for feature mapping and acoustic modeling, optimized for reverberant environments, and an online model selection method based on RT60 estimation.

Findings

Significant accuracy improvements over baseline systems.

Effective model selection using online RT60 estimation.

Robust performance across various reverberant conditions.

Abstract

Distant speech recognition is a challenge, particularly due to the corruption of speech signals by reverberation caused by large distances between the speaker and microphone. In order to cope with a wide range of reverberations in real-world situations, we present novel approaches for acoustic modeling including an ensemble of deep neural networks (DNNs) and an ensemble of jointly trained DNNs. First, multiple DNNs are established, each of which corresponds to a different reverberation time 60 (RT60) in a setup step. Also, each model in the ensemble of DNN acoustic models is further jointly trained, including both feature mapping and acoustic modeling, where the feature mapping is designed for the dereverberation as a front-end. In a testing phase, the two most likely DNNs are chosen from the DNN ensemble using maximum a posteriori (MAP) probabilities, computed in an online fashion by using maximum likelihood (ML)-based blind RT60 estimation and then the posterior probability outputs from two DNNs are combined using the ML-based weights as a simple average. Extensive experiments demonstrate that the proposed approach leads to substantial improvements in speech recognition accuracy over the conventional DNN baseline systems under diverse reverberant conditions.