Generative Adversarial Training Data Adaptation for Very Low-resource Automatic Speech Recognition

TL;DR

This paper introduces a CycleGAN-based voice conversion method to adapt training speech data to test speakers, significantly improving ASR performance on low-resource endangered language corpora.

Contribution

It presents a novel speaker adaptation technique using non-parallel voice conversion to enhance ASR accuracy for endangered languages with limited data.

Findings

35-60% relative reduction in phone error rate on Ainu corpus

40% relative reduction in phone error rate on Mboshi corpus

Outperforms conventional unsupervised and multilingual training methods

Abstract

It is important to transcribe and archive speech data of endangered languages for preserving heritages of verbal culture and automatic speech recognition (ASR) is a powerful tool to facilitate this process. However, since endangered languages do not generally have large corpora with many speakers, the performance of ASR models trained on them are considerably poor in general. Nevertheless, we are often left with a lot of recordings of spontaneous speech data that have to be transcribed. In this work, for mitigating this speaker sparsity problem, we propose to convert the whole training speech data and make it sound like the test speaker in order to develop a highly accurate ASR system for this speaker. For this purpose, we utilize a CycleGAN-based non-parallel voice conversion technology to forge a labeled training data that is close to the test speaker's speech. We evaluated this speaker adaptation approach on two low-resource corpora, namely, Ainu and Mboshi. We obtained 35-60% relative improvement in phone error rate on the Ainu corpus, and 40% relative improvement was attained on the Mboshi corpus. This approach outperformed two conventional methods namely unsupervised adaptation and multilingual training with these two corpora.