Exploiting Nontrivial Connectivity for Automatic Speech Recognition

TL;DR

This paper compares different deep network architectures and demonstrates their effective application to automatic speech recognition, leading to significant performance improvements.

Contribution

It introduces the application of residual, densely-connected, and highway networks to speech recognition, showing their advantages over traditional models.

Findings

Residual, dense, and highway networks improve speech recognition accuracy.

Deep networks with nontrivial connectivity address vanishing gradients.

Significant performance gains over existing models in speech recognition.

Abstract

Nontrivial connectivity has allowed the training of very deep networks by addressing the problem of vanishing gradients and offering a more efficient method of reusing parameters. In this paper we make a comparison between residual networks, densely-connected networks and highway networks on an image classification task. Next, we show that these methodologies can easily be deployed into automatic speech recognition and provide significant improvements to existing models.