Transformer Based Deliberation for Two-Pass Speech Recognition

TL;DR

This paper introduces a transformer-based deliberation model for two-pass speech recognition, improving accuracy and efficiency over LSTM-based methods by attending to audio and hypothesis text simultaneously.

Contribution

It replaces LSTM layers with transformer layers in deliberation rescoring, achieving better accuracy and lower computational cost.

Findings

7% relative word error rate improvement

38% reduction in computation

9% relative improvement over non-deliberation transformer rescoring

Abstract

Interactive speech recognition systems must generate words quickly while also producing accurate results. Two-pass models excel at these requirements by employing a first-pass decoder that quickly emits words, and a second-pass decoder that requires more context but is more accurate. Previous work has established that a deliberation network can be an effective second-pass model. The model attends to two kinds of inputs at once: encoded audio frames and the hypothesis text from the first-pass model. In this work, we explore using transformer layers instead of long-short term memory (LSTM) layers for deliberation rescoring. In transformer layers, we generalize the "encoder-decoder" attention to attend to both encoded audio and first-pass text hypotheses. The output context vectors are then combined by a merger layer. Compared to LSTM-based deliberation, our best transformer deliberation achieves 7% relative word error rate improvements along with a 38% reduction in computation. We also compare against non-deliberation transformer rescoring, and find a 9% relative improvement.