EAT: Enhanced ASR-TTS for Self-supervised Speech Recognition

TL;DR

This paper introduces EAT, an enhanced self-supervised speech model that improves out-of-domain ASR performance by incorporating language model rewards and attention scaling, significantly narrowing the gap with supervised methods.

Contribution

The paper proposes EAT, a novel self-supervised ASR-TTS model with two key features to better handle out-of-domain data, advancing self-supervised speech recognition.

Findings

EAT reduces the performance gap by over 2.6% on Librispeech.

EAT improves out-of-domain robustness in self-supervised ASR.

Training strategies enhance EAT's effectiveness under challenging conditions.

Abstract

Self-supervised ASR-TTS models suffer in out-of-domain data conditions. Here we propose an enhanced ASR-TTS (EAT) model that incorporates two main features: 1) The ASR$\rightarrow$TTS direction is equipped with a language model reward to penalize the ASR hypotheses before forwarding it to TTS. 2) In the TTS$\rightarrow$ASR direction, a hyper-parameter is introduced to scale the attention context from synthesized speech before sending it to ASR to handle out-of-domain data. Training strategies and the effectiveness of the EAT model are explored under out-of-domain data conditions. The results show that EAT reduces the performance gap between supervised and self-supervised training significantly by absolute 2.6\% and 2.7\% on Librispeech and BABEL respectively.