Speech representation learning: Learning bidirectional encoders with single-view, multi-view, and multi-task methods

TL;DR

This thesis explores various methods for speech representation learning across multiple settings, emphasizing the use of unlabeled and weakly labeled data to improve sequence prediction tasks before the Transformer era.

Contribution

It provides a comprehensive study of speech representation learning methods across supervised, unsupervised, semi-supervised, and multi-view settings, highlighting approaches beyond the Transformer-based models.

Findings

Effective multi-view learning strategies for speech data

Unsupervised and semi-supervised methods improve downstream tasks

Insights into speech representation learning before large-scale pre-training

Abstract

This thesis focuses on representation learning for sequence data over time or space, aiming to improve downstream sequence prediction tasks by using the learned representations. Supervised learning has been the most dominant approach for training deep neural networks for learning good sequential representations. However, one limiting factor to scale supervised learning is the lack of enough annotated data. Motivated by this challenge, it is natural to explore representation learning methods that can utilize large amounts of unlabeled and weakly labeled data, as well as an additional data modality. I describe my broad study of representation learning for speech data. Unlike most other works that focus on a single learning setting, this thesis studies multiple settings: supervised learning with auxiliary losses, unsupervised learning, semi-supervised learning, and multi-view learning. Besides different learning problems, I also explore multiple approaches for representation learning. Though I focus on speech data, the methods described in this thesis can also be applied to other domains. Overall, the field of representation learning is developing rapidly. State-of-the-art results on speech related tasks are typically based on Transformers pre-trained with large-scale self-supervised learning, which aims to learn generic representations that can benefit multiple downstream tasks. Since 2020, large-scale pre-training has been the de facto choice to achieve good performance. This delayed thesis does not attempt to summarize and compare with the latest results on speech representation learning; instead, it presents a unique study on speech representation learning before the Transformer era, that covers multiple learning settings. Some of the findings in this thesis can still be useful today.