Enhancing Speech Recognition Decoding via Layer Aggregation

TL;DR

This paper proposes a layer aggregation method for speech recognition models to improve decoding accuracy by leveraging information from intermediate layers, resulting in significant reductions in word and character error rates.

Contribution

It introduces a novel layer aggregation approach that combines multiple layers' predictions to enhance beam search decoding in speech recognition.

Findings

Layer aggregation improves WER and CER by up to 10% and 22%.

Intermediate layers contain useful information beyond top layers.

Aggregating layers relaxes model confidence and enhances decoding performance.

Abstract

Recently proposed speech recognition systems are designed to predict using representations generated by their top layers, employing greedy decoding which isolates each timestep from the rest of the sequence. Aiming for improved performance, a beam search algorithm is frequently utilized and a language model is incorporated to assist with ranking the top candidates. In this work, we experiment with several speech recognition models and find that logits predicted using the top layers may hamper beam search from achieving optimal results. Specifically, we show that fined-tuned Wav2Vec 2.0 and HuBERT yield highly confident predictions, and hypothesize that the predictions are based on local information and may not take full advantage of the information encoded in intermediate layers. To this end, we perform a layer analysis to reveal and visualize how predictions evolve throughout the inference flow. We then propose a prediction method that aggregates the top M layers, potentially leveraging useful information encoded in intermediate layers and relaxing model confidence. We showcase the effectiveness of our approach via beam search decoding, conducting our experiments on Librispeech test and dev sets and achieving WER, and CER reduction of up to 10% and 22%, respectively.