Echo State Speech Recognition

TL;DR

This paper introduces echo state network-inspired ASR models with partially untrained RNN components, showing they can match traditional models' performance while enabling more efficient training and storage.

Contribution

It demonstrates that randomizing certain RNN layers in ASR models does not degrade performance and can improve training efficiency, challenging conventional training practices.

Findings

Randomized decoders do not reduce model quality.

Encoder randomization negatively impacts performance.

ESN-inspired models match traditional models in accuracy.

Abstract

We propose automatic speech recognition (ASR) models inspired by echo state network (ESN), in which a subset of recurrent neural networks (RNN) layers in the models are randomly initialized and untrained. Our study focuses on RNN-T and Conformer models, and we show that model quality does not drop even when the decoder is fully randomized. Furthermore, such models can be trained more efficiently as the decoders do not require to be updated. By contrast, randomizing encoders hurts model quality, indicating that optimizing encoders and learn proper representations for acoustic inputs are more vital for speech recognition. Overall, we challenge the common practice of training ASR models for all components, and demonstrate that ESN-based models can perform equally well but enable more efficient training and storage than fully-trainable counterparts.