Lamer-SSL: Layer-aware Mixture of LoRA Experts for Continual Multilingual Expansion of Self-supervised Models without Forgetting

TL;DR

Lamer-SSL is a parameter-efficient framework that enables continual multilingual expansion of self-supervised speech models by balancing shared and language-specific representations, while mitigating forgetting through replay strategies.

Contribution

It introduces a layer-aware mixture of LoRA experts combined with replay to effectively extend models to new languages without losing prior knowledge.

Findings

Effective multilingual expansion on ASR and LID tasks.

Maintains performance on previous languages with minimal parameter updates.

Only 2.14% of parameters are trained during adaptation.

Abstract

Despite their impressive performance, self-supervised speech models often struggle to generalize to new languages and tend to forget previously acquired knowledge during continual training. To address this, we propose Lamer-SSL, a parameter-efficient framework that integrates a Layer-Aware MixturE of LoRA Experts (Lamer) module with a replay strategy. The Lamer module enables flexible balancing between shared and language-specific representations, while layer-aware expert allocation assigns more experts to deeper layers where semantic information is richer. Meanwhile, the replay strategy retains prior knowledge using minimal data, mitigating forgetting during continual training. Experiments on automatic speech recognition (ASR) and language identification (LID) demonstrate that Lamer-SSL extends self-supervised models to new languages effectively while maintaining strong performance on previously learned languages with only 2.14% parameters being trainable.