SSVD-O: Parameter-Efficient Fine-Tuning with Structured SVD for Speech Recognition

TL;DR

This paper introduces SSVD-O, a structured SVD-based parameter-efficient fine-tuning method for speech recognition that improves adaptation efficiency, balances learning and forgetting, and outperforms existing PEFT methods on domain-shifted ASR tasks.

Contribution

The paper presents SSVD-O, a novel structured SVD-guided fine-tuning approach that enhances speech model adaptation by balancing parameter allocation and reducing forgetting.

Findings

SSVD-O narrows the performance gap to full fine-tuning.

SSVD-O improves generalization on domain-shifted speech tasks.

SSVD-O mitigates catastrophic forgetting in speech model adaptation.

Abstract

Parameter-efficient fine-tuning (PEFT) is a scalable approach for adapting large speech foundation models to new domains. While methods such as LoRA and its state-of-the-art variants reduce adaptation costs, they typically allocate parameters uniformly across model subspaces, which limits their efficiency and scalability in speech applications. Building on our prior work, this paper introduces SSVD-Outer (SSVD-O), an extension of the structured SVD-guided (SSVD) fine-tuning method. SSVD-O combines input acoustic feature space-associated inner transformations with output semantic feature space-associated outer transformations to enable scalable and balanced adaptation. We conduct the first systematic analysis of parameter budget allocation across model subspaces in PEFT for automatic speech recognition (ASR), and investigate the trade-off between learning and forgetting under constrained resources. SSVD-O is benchmarked against LoRA, DoRA, PiSSA, and SSVD on domain-shifted ASR tasks, including child speech and regional accents, across model scales from 0.1B to 2B within the ESPnet framework. Experimental results show that SSVD-O consistently narrows the performance gap to full fine-tuning while improving generalization and mitigating catastrophic forgetting.