A Distributed Optimisation Framework Combining Natural Gradient with Hessian-Free for Discriminative Sequence Training

TL;DR

This paper introduces a distributed optimization framework combining natural gradient and Hessian-free methods, significantly improving training efficiency and accuracy for neural networks in speech recognition tasks.

Contribution

The novel NGHF framework effectively combines natural gradient and Hessian-free methods with a new CG solution and preconditioning, enabling efficient distributed training with fewer iterations.

Findings

NGHF outperforms SGD and Adam in word error rate reduction

Requires 5-8 CG iterations instead of 200

Achieves larger WER reductions with fewer parameter updates

Abstract

This paper presents a novel natural gradient and Hessian-free (NGHF) optimisation framework for neural network training that can operate efficiently in a distributed manner. It relies on the linear conjugate gradient (CG) algorithm to combine the natural gradient (NG) method with local curvature information from Hessian-free (HF) or other second-order methods. A solution to a numerical issue in CG allows effective parameter updates to be generated with far fewer CG iterations than usually used (e.g. 5-8 instead of 200). This work also presents a novel preconditioning approach to improve the progress made by individual CG iterations for models with shared parameters. Although applicable to other training losses and model structures, NGHF is investigated in this paper for lattice-based discriminative sequence training for hybrid hidden Markov model acoustic models using a standard recurrent neural network, long short-term memory, and time delay neural network models for output probability calculation. Automatic speech recognition experiments are reported on the multi-genre broadcast data set for a range of different acoustic model types. These experiments show that NGHF achieves larger word error rate reductions than standard stochastic gradient descent or Adam, while requiring orders of magnitude fewer parameter updates.