Scaling Up Deliberation for Multilingual ASR

TL;DR

This paper explores second-pass multilingual deliberation in end-to-end speech recognition, demonstrating significant WER improvements across multiple languages by scaling the deliberation components and utilizing transformer-based rescoring.

Contribution

It introduces a scalable multilingual deliberation framework with a transformer-based rescoring method, enhancing speech recognition accuracy over single-pass models.

Findings

Deliberation improves average WER by 4% across 9 languages.

Scaling the deliberation model up to 1B parameters yields a 9% WER reduction.

Transformer-based rescoring allows parallelization during inference.

Abstract

Multilingual end-to-end automatic speech recognition models are attractive due to its simplicity in training and deployment. Recent work on large-scale training of such models has shown promising results compared to monolingual models. However, the work often focuses on multilingual models themselves in a single-pass setup. In this work, we investigate second-pass deliberation for multilingual speech recognition. Our proposed deliberation is multilingual, i.e., the text encoder encodes hypothesis text from multiple languages, and the decoder attends to multilingual text and audio. We investigate scaling the deliberation text encoder and decoder, and compare scaling the deliberation decoder and the first-pass cascaded encoder. We show that deliberation improves the average WER on 9 languages by 4% relative compared to the single-pass model. By increasing the size of the deliberation up to 1B parameters, the average WER improvement increases to 9%, with up to 14% for certain languages. Our deliberation rescorer is based on transformer layers and can be parallelized during rescoring.