Residual Adapters for Parameter-Efficient ASR Adaptation to Atypical and Accented Speech

TL;DR

This paper introduces residual adapters that enable efficient adaptation of ASR systems to atypical and accented speech by adding minimal parameters, achieving comparable performance to full model fine-tuning.

Contribution

The authors propose residual adapters for ASR models, allowing effective speaker adaptation with less than 0.5% of parameters, improving scalability and efficiency.

Findings

Residual adapters match fine-tuning performance in speech adaptation.

Less than 0.5% of parameters are updated during adaptation.

Effective on multiple speech adaptation tasks and architectures.

Abstract

Automatic Speech Recognition (ASR) systems are often optimized to work best for speakers with canonical speech patterns. Unfortunately, these systems perform poorly when tested on atypical speech and heavily accented speech. It has previously been shown that personalization through model fine-tuning substantially improves performance. However, maintaining such large models per speaker is costly and difficult to scale. We show that by adding a relatively small number of extra parameters to the encoder layers via so-called residual adapter, we can achieve similar adaptation gains compared to model fine-tuning, while only updating a tiny fraction (less than 0.5%) of the model parameters. We demonstrate this on two speech adaptation tasks (atypical and accented speech) and for two state-of-the-art ASR architectures.