Bayesian Low-Rank Factorization for Robust Model Adaptation

TL;DR

This paper introduces Bayesian factorized adapters for speech models, enabling effective domain adaptation like code-switching with minimal loss of general performance and reduced catastrophic forgetting.

Contribution

It proposes a Bayesian approach to low-rank adaptation matrices that improves domain adaptation in speech models while preserving their broad capabilities.

Findings

Achieves 54% backward gain over LoRA

Only 4% performance drop on new domain

Minimal adaptation loss and reduced forgetting

Abstract

Large speech foundation models achieve strong performance across many domains, but they often require adaptation to handle local needs such as code-switching, where speakers mix languages within the same utterance. Direct fine-tuning of these models risks overfitting to the target domain and overwriting the broad capabilities of the base model. To address this challenge, we explore Bayesian factorized adapters for speech foundation models, which place priors near zero to achieve sparser adaptation matrices and thereby retain general performance while adapting to specific domains. We apply our approach to the Whisper model and evaluate on different multilingual code-switching scenarios. Our results show only minimal adaptation loss while significantly reducing catastrophic forgetting of the base model. Compared to LoRA, our method achieves a backward gain of 54% with only a 4% drop on the new domain. These findings highlight the effectiveness of Bayesian adaptation for fine-tuning speech foundation models without sacrificing generalization.