Multilingual and crosslingual speech recognition using phonological-vector based phone embeddings

TL;DR

This paper introduces JoinAP, a novel method combining phonology-driven phone embeddings with neural acoustic features, enabling effective multilingual and crosslingual speech recognition without acoustic-to-phonological inversion.

Contribution

It proposes a top-down phonological-vector based phone embedding method integrated with DNN acoustic features, improving multilingual and crosslingual speech recognition performance.

Findings

JoinAP with nonlinear embeddings outperforms linear and traditional methods.

Effective zero-shot and few-shot crosslingual recognition demonstrated.

Method achieves superior results on multilingual datasets.

Abstract

The use of phonological features (PFs) potentially allows language-specific phones to remain linked in training, which is highly desirable for information sharing for multilingual and crosslingual speech recognition methods for low-resourced languages. A drawback suffered by previous methods in using phonological features is that the acoustic-to-PF extraction in a bottom-up way is itself difficult. In this paper, we propose to join phonology driven phone embedding (top-down) and deep neural network (DNN) based acoustic feature extraction (bottom-up) to calculate phone probabilities. The new method is called JoinAP (Joining of Acoustics and Phonology). Remarkably, no inversion from acoustics to phonological features is required for speech recognition. For each phone in the IPA (International Phonetic Alphabet) table, we encode its phonological features to a phonological-vector, and then apply linear or nonlinear transformation of the phonological-vector to obtain the phone embedding. A series of multilingual and crosslingual (both zero-shot and few-shot) speech recognition experiments are conducted on the CommonVoice dataset (German, French, Spanish and Italian) and the AISHLL-1 dataset (Mandarin), and demonstrate the superiority of JoinAP with nonlinear phone embeddings over both JoinAP with linear phone embeddings and the traditional method with flat phone embeddings.