GMM-Free Flat Start Sequence-Discriminative DNN Training

TL;DR

This paper introduces a GMM-free, sequence-discriminative flat start training method for DNN-based HMMs that is faster and yields slightly better word error rates compared to traditional iterative approaches.

Contribution

It proposes a novel GMM-free flat start training approach using sequence-discriminative criteria and KL-divergence-based state clustering, improving speed and accuracy.

Findings

Faster training than iterative realignment methods

Slightly improved word error rates

Effective GMM-free training process

Abstract

Recently, attempts have been made to remove Gaussian mixture models (GMM) from the training process of deep neural network-based hidden Markov models (HMM/DNN). For the GMM-free training of a HMM/DNN hybrid we have to solve two problems, namely the initial alignment of the frame-level state labels and the creation of context-dependent states. Although flat-start training via iteratively realigning and retraining the DNN using a frame-level error function is viable, it is quite cumbersome. Here, we propose to use a sequence-discriminative training criterion for flat start. While sequence-discriminative training is routinely applied only in the final phase of model training, we show that with proper caution it is also suitable for getting an alignment of context-independent DNN models. For the construction of tied states we apply a recently proposed KL-divergence-based state clustering method, hence our whole training process is GMM-free. In the experimental evaluation we found that the sequence-discriminative flat start training method is not only significantly faster than the straightforward approach of iterative retraining and realignment, but the word error rates attained are slightly better as well.