Combining Spectral and Self-Supervised Features for Low Resource Speech Recognition and Translation

TL;DR

This paper explores combining spectral features with self-supervised learning representations to improve low-resource speech recognition and translation, demonstrating significant performance gains and analyzing the impact of domain mismatch.

Contribution

It introduces a learnable, interpretable framework for combining spectral features and SSL representations, and proposes a mixture of experts model to address domain mismatch issues.

Findings

Combined spectral and SSL features outperform baselines in low-resource tasks.

The mixture of experts model shows SSL's limited contribution under domain mismatch.

Spectral features provide robustness to domain shifts in speech tasks.

Abstract

Self-Supervised Learning (SSL) models have been successfully applied in various deep learning-based speech tasks, particularly those with a limited amount of data. However, the quality of SSL representations depends highly on the relatedness between the SSL training domain(s) and the target data domain. On the contrary, spectral feature (SF) extractors such as log Mel-filterbanks are hand-crafted non-learnable components, and could be more robust to domain shifts. The present work examines the assumption that combining non-learnable SF extractors to SSL models is an effective approach to low resource speech tasks. We propose a learnable and interpretable framework to combine SF and SSL representations. The proposed framework outperforms significantly both baseline and SSL models on Automatic Speech Recognition (ASR) and Speech Translation (ST) tasks on three low resource datasets. We additionally design a mixture of experts based combination model. This last model reveals that the relative contribution of SSL models over conventional SF extractors is very small in case of domain mismatch between SSL training set and the target language data.