VARAN: Variational Inference for Self-Supervised Speech Models Fine-Tuning on Downstream Tasks

TL;DR

VARAN introduces a dynamic layer aggregation framework for self-supervised speech models, improving task performance by adaptively weighting layer features based on individual inputs, especially when combined with LoRA fine-tuning.

Contribution

It presents a novel input-dependent layer aggregation method using probing heads, enhancing the adaptation of self-supervised speech models for downstream tasks.

Findings

Superior performance on speech recognition and emotion recognition tasks.

Effective when combined with LoRA fine-tuning.

Resolves information bottleneck issues in layer aggregation.

Abstract

Conventional methods for aggregating layers in fine-tuned self-supervised speech models, such as using the final layer or weighted sum, suffer from information bottlenecks and static feature weighting for all dataset examples. We propose VARAN, a framework that dynamically tailors layer aggregation to individual inputs. By employing layer-specialized probing heads and data-dependent weighting, VARAN adaptively prioritizes layer's features based on input. Evaluations on automatic speech recognition and speech emotion recognition tasks demonstrate VARAN's superior performance, particularly when using the LoRA fine-tuning technique. The framework resolves the trade-off between preserving layer-specific information and enabling flexible feature utilization, advancing efficient adaptation of self-supervised speech representations.