Task Arithmetic with Support Languages for Low-Resource ASR

TL;DR

This paper introduces a method for improving low-resource automatic speech recognition by combining task-specific models trained on related languages, resulting in significant WER reductions.

Contribution

It proposes a novel task arithmetic approach that merges models from high- and low-resource languages to enhance low-resource ASR performance.

Findings

Up to 10% WER reduction across 23 low-resource languages.

Consistent improvements over baseline models.

Effective linear combination of task vectors for low-resource ASR.

Abstract

The development of resource-constrained approaches to automatic speech recognition (ASR) is of great interest due to its broad applicability to many low-resource languages for which there is scant usable data. Existing approaches to many low-resource natural language processing tasks leverage additional data from higher-resource languages that are closely related to a target low-resource language. One increasingly popular approach uses task arithmetic to combine models trained on different tasks to create a model for a task where there is little to no training data. In this paper, we consider training on a particular language to be a task, and we generate task vectors by fine-tuning variants of the Whisper ASR system. For pairs of high- and low-resource languages, we merge task vectors via a linear combination which is optimized on the downstream word error rate on the low-resource target language's validation set. Across 23 low-resource target languages for which we evaluate this technique, we find consistent word error rate improvements of up to 10% compared to a baseline without our approach.