Domain Adaptation of low-resource Target-Domain models using well-trained ASR Conformer Models

TL;DR

This paper presents a domain adaptation method for low-resource ASR that leverages embeddings from well-trained models' encoder layers, combined with Spectral Augmentation, to significantly improve target-domain recognition performance.

Contribution

The study introduces a novel approach of using encoder layer embeddings from pre-trained ASR models for target-domain adaptation, enhancing low-resource ASR performance.

Findings

30% relative improvement on LibriSpeech-100-clean data

50% relative improvement on WSJ data

Effective combination of encoder embeddings and Spectral Augmentation

Abstract

In this paper, we investigate domain adaptation for low-resource Automatic Speech Recognition (ASR) of target-domain data, when a well-trained ASR model trained with a large dataset is available. We argue that in the encoder-decoder framework, the decoder of the well-trained ASR model is largely tuned towards the source-domain, hurting the performance of target-domain models in vanilla transfer-learning. On the other hand, the encoder layers of the well-trained ASR model mostly capture the acoustic characteristics. We, therefore, propose to use the embeddings tapped from these encoder layers as features for a downstream Conformer target-domain model and show that they provide significant improvements. We do ablation studies on which encoder layer is optimal to tap the embeddings, as well as the effect of freezing or updating the well-trained ASR model's encoder layers. We further show that applying Spectral Augmentation (SpecAug) on the proposed features (this is in addition to default SpecAug on input spectral features) provides a further improvement on the target-domain performance. For the LibriSpeech-100-clean data as target-domain and SPGI-5000 as a well-trained model, we get 30% relative improvement over baseline. Similarly, with WSJ data as target-domain and LibriSpeech-960 as a well-trained model, we get 50% relative improvement over baseline.