Unsupervised Model-based speaker adaptation of end-to-end lattice-free MMI model for speech recognition

TL;DR

This paper introduces an unsupervised speaker adaptation method for end-to-end lattice-free MMI speech recognition models using LHUC and BLHUC techniques, significantly reducing word error rates on the Switchboard dataset.

Contribution

It proposes a novel unsupervised model-based adaptation framework for E2E LF-MMI models employing LHUC/BLHUC, with systematic regularization and confidence-based data selection.

Findings

BLHUC adaptation reduces WER by up to 14.7% relative.

The proposed method achieves WERs comparable to state-of-the-art hybrid and Conformer systems.

Confidence score-based data selection improves adaptation effectiveness.

Abstract

Modeling the speaker variability is a key challenge for automatic speech recognition (ASR) systems. In this paper, the learning hidden unit contributions (LHUC) based adaptation techniques with compact speaker dependent (SD) parameters are used to facilitate both speaker adaptive training (SAT) and unsupervised test-time speaker adaptation for end-to-end (E2E) lattice-free MMI (LF-MMI) models. An unsupervised model-based adaptation framework is proposed to estimate the SD parameters in E2E paradigm using LF-MMI and cross entropy (CE) criterions. Various regularization methods of the standard LHUC adaptation, e.g., the Bayesian LHUC (BLHUC) adaptation, are systematically investigated to mitigate the risk of overfitting, on E2E LF-MMI CNN-TDNN and CNN-TDNN-BLSTM models. Lattice-based confidence score estimation is used for adaptation data selection to reduce the supervision label uncertainty. Experiments on the 300-hour Switchboard task suggest that applying BLHUC in the proposed unsupervised E2E adaptation framework to byte pair encoding (BPE) based E2E LF-MMI systems consistently outperformed the baseline systems by relative word error rate (WER) reductions up to 10.5% and 14.7% on the NIST Hub5'00 and RT03 evaluation sets, and achieved the best performance in WERs of 9.0% and 9.7%, respectively. These results are comparable to the results of state-of-the-art adapted LF-MMI hybrid systems and adapted Conformer-based E2E systems.