A Robust Incomplete Multimodal Low-Rank Adaptation Approach for Emotion Recognition

TL;DR

This paper introduces MCULoRA, a novel, parameter-efficient framework for emotion recognition from incomplete multimodal data, effectively decoupling modality information and dynamically optimizing training for better accuracy.

Contribution

The paper proposes a new low-rank adaptation method with modules for decoupling shared and unique modality features and dynamically tuning training, addressing conflicts in previous approaches.

Findings

MCULoRA outperforms existing methods in benchmark datasets.

It effectively decouples shared and individual modality information.

Dynamic parameter fine-tuning improves training efficiency.

Abstract

Multimodal Emotion Recognition (MER) often encounters incomplete multimodality in practical applications due to sensor failures or privacy protection requirements. While existing methods attempt to address various incomplete multimodal scenarios by balancing the training of each modality combination through additional gradients, these approaches face a critical limitation: training gradients from different modality combinations conflict with each other, ultimately degrading the performance of the final prediction model. In this paper, we propose a unimodal decoupled dynamic low-rank adaptation method based on modality combinations, named MCULoRA, which is a novel framework for the parameter-efficient training of incomplete multimodal learning models. MCULoRA consists of two key modules, modality combination aware low-rank adaptation (MCLA) and dynamic parameter fine-tuning (DPFT). The MCLA module effectively decouples the shared information from the distinct characteristics of individual modality combinations. The DPFT module adjusts the training ratio of modality combinations based on the separability of each modality's representation space, optimizing the learning efficiency across different modality combinations. Our extensive experimental evaluation in multiple benchmark datasets demonstrates that MCULoRA substantially outperforms previous incomplete multimodal learning approaches in downstream task accuracy.