Sequence-to-Sequence ASR Optimization via Reinforcement Learning

TL;DR

This paper introduces a reinforcement learning-based training method for sequence-to-sequence speech recognition models, directly optimizing recognition accuracy and reducing errors compared to traditional likelihood-based training.

Contribution

It proposes using policy gradient reinforcement learning with negative Levenshtein distance as reward to improve sequence-to-sequence ASR performance.

Findings

Significant performance improvements over maximum likelihood training.

Effective direct optimization of recognition error metrics.

Demonstrated robustness in transcription accuracy.

Abstract

Despite the success of sequence-to-sequence approaches in automatic speech recognition (ASR) systems, the models still suffer from several problems, mainly due to the mismatch between the training and inference conditions. In the sequence-to-sequence architecture, the model is trained to predict the grapheme of the current time-step given the input of speech signal and the ground-truth grapheme history of the previous time-steps. However, it remains unclear how well the model approximates real-world speech during inference. Thus, generating the whole transcription from scratch based on previous predictions is complicated and errors can propagate over time. Furthermore, the model is optimized to maximize the likelihood of training data instead of error rate evaluation metrics that actually quantify recognition quality. This paper presents an alternative strategy for training sequence-to-sequence ASR models by adopting the idea of reinforcement learning (RL). Unlike the standard training scheme with maximum likelihood estimation, our proposed approach utilizes the policy gradient algorithm. We can (1) sample the whole transcription based on the model's prediction in the training process and (2) directly optimize the model with negative Levenshtein distance as the reward. Experimental results demonstrate that we significantly improved the performance compared to a model trained only with maximum likelihood estimation.