Using External Off-Policy Speech-To-Text Mappings in Contextual End-To-End Automated Speech Recognition

TL;DR

This paper proposes a method to improve speech recognition models by using external knowledge from text-to-speech generated embeddings, enabling faster adaptation and better accuracy in new domains.

Contribution

It introduces a novel approach that leverages off-policy key-value stores with text-to-speech embeddings for efficient domain adaptation in ASR systems.

Findings

Reduces domain adaptation time by up to 1,000 GPU-hours.

Achieves up to 3% WER improvement over fine-tuning baseline.

Effective in zero and few-shot adaptation scenarios.

Abstract

Despite improvements to the generalization performance of automated speech recognition (ASR) models, specializing ASR models for downstream tasks remains a challenging task, primarily due to reduced data availability (necessitating increased data collection), and rapidly shifting data distributions (requiring more frequent model fine-tuning). In this work, we investigate the potential of leveraging external knowledge, particularly through off-policy key-value stores generated with text-to-speech methods, to allow for flexible post-training adaptation to new data distributions. In our approach, audio embeddings captured from text-to-speech, along with semantic text embeddings, are used to bias ASR via an approximate k-nearest-neighbor (KNN) based attentive fusion step. Our experiments on LibiriSpeech and in-house voice assistant/search datasets show that the proposed approach can reduce domain adaptation time by up to 1K GPU-hours while providing up to 3% WER improvement compared to a fine-tuning baseline, suggesting a promising approach for adapting production ASR systems in challenging zero and few-shot scenarios.