Learning Phone Recognition from Unpaired Audio and Phone Sequences Based on Generative Adversarial Network

TL;DR

This paper presents a novel two-stage GAN-based framework for learning phone recognition directly from unpaired speech and phone sequences, reducing reliance on large paired datasets.

Contribution

It introduces a new iterative approach combining GANs and HMMs to improve phone recognition from unpaired data, outperforming existing methods.

Findings

Outperforms acoustic unit discovery methods on TIMIT dataset

Effectively learns from unpaired speech and phone sequences

Enhances segmentation accuracy for phone recognition

Abstract

ASR has been shown to achieve great performance recently. However, most of them rely on massive paired data, which is not feasible for low-resource languages worldwide. This paper investigates how to learn directly from unpaired phone sequences and speech utterances. We design a two-stage iterative framework. GAN training is adopted in the first stage to find the mapping relationship between unpaired speech and phone sequence. In the second stage, another HMM model is introduced to train from the generator's output, which boosts the performance and provides a better segmentation for the next iteration. In the experiment, we first investigate different choices of model designs. Then we compare the framework to different types of baselines: (i) supervised methods (ii) acoustic unit discovery based methods (iii) methods learning from unpaired data. Our framework performs consistently better than all acoustic unit discovery methods and previous methods learning from unpaired data based on the TIMIT dataset.