AdaLTM: Adaptive Layer-wise Task Vector Merging for Categorical Speech Emotion Recognition with ASR Knowledge Integration

TL;DR

This paper introduces AdaLTM, a novel method for integrating ASR and SER tasks by layer-wise merging of task vectors, improving emotion recognition by balancing linguistic and paralinguistic information without gradient conflicts.

Contribution

The paper proposes a new layer-wise task vector merging framework that leverages frozen models and learnable coefficients to enhance speech emotion recognition with ASR knowledge.

Findings

Effective mitigation of conflicts between ASR and SER.

Improved emotion recognition accuracy on MSP-Podcast.

Layer-wise integration balances linguistic and emotional features.

Abstract

Integrating Automatic Speech Recognition (ASR) into Speech Emotion Recognition (SER) enhances modeling by providing linguistic context. However, conventional feature fusion faces performance bottlenecks, and multi-task learning often suffers from optimization conflicts. While task vectors and model merging have addressed such conflicts in NLP and CV, their potential in speech tasks remains largely unexplored. In this work, we propose an Adaptive Layer-wise Task Vector Merging (AdaLTM) framework based on WavLM-Large. Instead of joint optimization, we extract task vectors from in-domain ASR and SER models fine-tuned on emotion datasets. These vectors are integrated into a frozen base model using layer-wise learnable coefficients. This strategy enables depth-aware balancing of linguistic and paralinguistic knowledge across transformer layers without gradient interference. Experiments on the MSP-Podcast demonstrate that the proposed approach effectively mitigates conflicts between ASR and SER.