Bi-APC: Bidirectional Autoregressive Predictive Coding for Unsupervised Pre-training and Its Application to Children's ASR

TL;DR

This paper introduces Bi-APC, a bidirectional unsupervised pre-training method for speech recognition, demonstrating its effectiveness in improving children's ASR by transferring knowledge from adult speech data.

Contribution

The paper proposes Bi-APC, extending autoregressive predictive coding to bidirectional pre-training, and applies it to enhance children's speech recognition models.

Findings

Bi-APC achieves comparable performance to supervised pre-training on child speech.

Unsupervised pre-training with Bi-APC benefits bidirectional LSTM models.

Bi-APC improves WER over baseline models.

Abstract

We present a bidirectional unsupervised model pre-training (UPT) method and apply it to children's automatic speech recognition (ASR). An obstacle to improving child ASR is the scarcity of child speech databases. A common approach to alleviate this problem is model pre-training using data from adult speech. Pre-training can be done using supervised (SPT) or unsupervised methods, depending on the availability of annotations. Typically, SPT performs better. In this paper, we focus on UPT to address the situations when pre-training data are unlabeled. Autoregressive predictive coding (APC), a UPT method, predicts frames from only one direction, limiting its use to uni-directional pre-training. Conventional bidirectional UPT methods, however, predict only a small portion of frames. To extend the benefits of APC to bi-directional pre-training, Bi-APC is proposed. We then use adaptation techniques to transfer knowledge learned from adult speech (using the Librispeech corpus) to child speech (OGI Kids corpus). LSTM-based hybrid systems are investigated. For the uni-LSTM structure, APC obtains similar WER improvements to SPT over the baseline. When applied to BLSTM, however, APC is not as competitive as SPT, but our proposed Bi-APC has comparable improvements to SPT.