Improving Semi-supervised End-to-end Automatic Speech Recognition using CycleGAN and Inter-domain Losses

TL;DR

This paper introduces a combined CycleGAN and inter-domain loss approach to enhance semi-supervised end-to-end speech recognition, effectively leveraging unpaired speech and text data to improve accuracy.

Contribution

It presents a novel method integrating CycleGAN and inter-domain losses for better shared representations in semi-supervised speech recognition.

Findings

Achieved 8-8.5% CER reduction on WSJ eval92 and Voxforge datasets.

Demonstrated noticeable improvements on LibriSpeech test_clean.

Validated effectiveness of combining CycleGAN with inter-domain loss.

Abstract

We propose a novel method that combines CycleGAN and inter-domain losses for semi-supervised end-to-end automatic speech recognition. Inter-domain loss targets the extraction of an intermediate shared representation of speech and text inputs using a shared network. CycleGAN uses cycle-consistent loss and the identity mapping loss to preserve relevant characteristics of the input feature after converting from one domain to another. As such, both approaches are suitable to train end-to-end models on unpaired speech-text inputs. In this paper, we exploit the advantages from both inter-domain loss and CycleGAN to achieve better shared representation of unpaired speech and text inputs and thus improve the speech-to-text mapping. Our experimental results on the WSJ eval92 and Voxforge (non English) show 8~8.5% character error rate reduction over the baseline, and the results on LibriSpeech test_clean also show noticeable improvement.