Attention-weighted Centered Kernel Alignment for Knowledge Distillation in Large Audio-Language Models Applied to Speech Emotion Recognition

TL;DR

This paper introduces PL-Distill, a novel knowledge distillation framework for large audio-language models in speech emotion recognition, utilizing attention-weighted kernel alignment for effective cross-modal feature alignment and model compression.

Contribution

It proposes a new attention-weighted centered kernel alignment method for better feature alignment in knowledge distillation of large audio-language models.

Findings

PL-Distill compresses models from 8.4B to 1.1B parameters.

It outperforms state-of-the-art models and baselines on multiple datasets.

The method effectively aligns cross-modal features despite dimension mismatches.

Abstract

The emergence of Large Audio-Language Models (LALMs) has advanced Speech Emotion Recognition (SER), but their size limits deployment in resource-constrained environments. While Knowledge Distillation is effective for LALM compression, existing methods remain underexplored in distilling the cross-modal projection module (Projector), and often struggle with alignment due to differences in feature dimensions. We propose PL-Distill, a KD framework that combines Projector-Level Distillation (PDist) to align audio embeddings and Logits-Level Distillation (LDist) to align output logits. PDist introduces Attention-weighted Centered Kernel Alignment, a novel approach we propose to highlight important time steps and address dimension mismatches. Meanwhile, LDist minimizes the Kullback-Leibler divergence between teacher and student logits from audio and text modalities. On IEMOCAP, RAVDESS, and SAVEE, PL-Distill compresses an 8.4B-parameter teacher to a compact 1.1B-parameter student, consistently outperforming the teacher, state-of-the-art pretrained models, and other KD baselines across all metrics.