SynthASR: Unlocking Synthetic Data for Speech Recognition

TL;DR

SynthASR demonstrates that synthetic speech can effectively train end-to-end speech recognition models, especially for new applications with limited data, reducing costs and dependency on real data.

Contribution

The paper introduces a novel multi-stage training strategy utilizing synthetic speech for E2E ASR models, addressing catastrophic forgetting and improving performance on new applications.

Findings

Over 65% relative improvement in recognizing medication names.

Effective training of large-scale E2E ASR models with synthetic data.

Reduced dependency on real, costly data for new application development.

Abstract

End-to-end (E2E) automatic speech recognition (ASR) models have recently demonstrated superior performance over the traditional hybrid ASR models. Training an E2E ASR model requires a large amount of data which is not only expensive but may also raise dependency on production data. At the same time, synthetic speech generated by the state-of-the-art text-to-speech (TTS) engines has advanced to near-human naturalness. In this work, we propose to utilize synthetic speech for ASR training (SynthASR) in applications where data is sparse or hard to get for ASR model training. In addition, we apply continual learning with a novel multi-stage training strategy to address catastrophic forgetting, achieved by a mix of weighted multi-style training, data augmentation, encoder freezing, and parameter regularization. In our experiments conducted on in-house datasets for a new application of recognizing medication names, training ASR RNN-T models with synthetic audio via the proposed multi-stage training improved the recognition performance on new application by more than 65% relative, without degradation on existing general applications. Our observations show that SynthASR holds great promise in training the state-of-the-art large-scale E2E ASR models for new applications while reducing the costs and dependency on production data.