Training variance and performance evaluation of neural networks in speech

TL;DR

This paper investigates the variability in neural network training outcomes for speech recognition, highlighting the significant impact of training variance and urging a reevaluation of result reporting practices.

Contribution

It provides the first extensive empirical analysis of training variance effects in speech recognition neural networks, emphasizing the need for more robust evaluation methods.

Findings

Training results exhibit substantial variance across configurations.

Training as sampling from a distribution impacts performance evaluation.

Results suggest rethinking reporting standards in speech recognition research.

Abstract

In this work we study variance in the results of neural network training on a wide variety of configurations in automatic speech recognition. Although this variance itself is well known, this is, to the best of our knowledge, the first paper that performs an extensive empirical study on its effects in speech recognition. We view training as sampling from a distribution and show that these distributions can have a substantial variance. These results show the urgent need to rethink the way in which results in the literature are reported and interpreted.