Acoustic feature learning using cross-domain articulatory measurements

TL;DR

This paper proposes methods for acoustic feature learning using cross-domain articulatory data, improving speech recognition across different datasets without requiring domain-specific articulatory measurements.

Contribution

It introduces deep variational CCA-based techniques for acoustic feature learning with domain-mismatched data, enhancing portability and accuracy in phonetic recognition.

Findings

Improved phonetic recognition on TIMIT and Wall Street Journal datasets.

Effective use of domain-mismatched articulatory data for feature learning.

Analysis of design choices impacting model performance.

Abstract

Previous work has shown that it is possible to improve speech recognition by learning acoustic features from paired acoustic-articulatory data, for example by using canonical correlation analysis (CCA) or its deep extensions. One limitation of this prior work is that the learned feature models are difficult to port to new datasets or domains, and articulatory data is not available for most speech corpora. In this work we study the problem of acoustic feature learning in the setting where we have access to an external, domain-mismatched dataset of paired speech and articulatory measurements, either with or without labels. We develop methods for acoustic feature learning in these settings, based on deep variational CCA and extensions that use both source and target domain data and labels. Using this approach, we improve phonetic recognition accuracies on both TIMIT and Wall Street Journal and analyze a number of design choices.