Decoupled Hierarchical Distillation for Multimodal Emotion Recognition

TL;DR

This paper introduces a hierarchical distillation framework for multimodal emotion recognition that decouples features into homogeneous and heterogeneous components, enabling effective cross-modal knowledge transfer and improved recognition accuracy.

Contribution

The proposed Decoupled Hierarchical Multimodal Distillation (DHMD) framework innovatively combines feature decoupling with two-stage knowledge distillation to enhance multimodal emotion recognition.

Findings

DHMD outperforms state-of-the-art MER methods on CMU-MOSI and CMU-MOSEI datasets.

The framework achieves up to 2.4% accuracy improvement over existing methods.

Visualization shows meaningful distribution patterns in modality-irrelevant and exclusive feature spaces.

Abstract

Human multimodal emotion recognition (MER) seeks to infer human emotions by integrating information from language, visual, and acoustic modalities. Although existing MER approaches have achieved promising results, they still struggle with inherent multimodal heterogeneities and varying contributions from different modalities. To address these challenges, we propose a novel framework, Decoupled Hierarchical Multimodal Distillation (DHMD). DHMD decouples each modality's features into modality-irrelevant (homogeneous) and modality-exclusive (heterogeneous) components using a self-regression mechanism. The framework employs a two-stage knowledge distillation (KD) strategy: (1) coarse-grained KD via a Graph Distillation Unit (GD-Unit) in each decoupled feature space, where a dynamic graph facilitates adaptive distillation among modalities, and (2) fine-grained KD through a cross-modal dictionary matching mechanism, which aligns semantic granularities across modalities to produce more discriminative MER representations. This hierarchical distillation approach enables flexible knowledge transfer and effectively improves cross-modal feature alignment. Experimental results demonstrate that DHMD consistently outperforms state-of-the-art MER methods, achieving 1.3\%/2.4\% (ACC$_7$), 1.3\%/1.9\% (ACC$_2$) and 1.9\%/1.8\% (F1) relative improvement on CMU-MOSI/CMU-MOSEI dataset, respectively. Meanwhile, visualization results reveal that both the graph edges and dictionary activations in DHMD exhibit meaningful distribution patterns across modality-irrelevant/-exclusive feature spaces.