Multimodal Mixture of Low-Rank Experts for Sentiment Analysis and Emotion Recognition

TL;DR

This paper introduces MMoLRE, a novel multi-task learning approach that employs shared and task-specific low-rank experts to improve multimodal sentiment analysis and emotion recognition, reducing parameter overhead and avoiding conflicts.

Contribution

The paper proposes a new MTL method with low-rank experts for better task modeling and efficiency in multimodal sentiment and emotion analysis.

Findings

Achieves state-of-the-art results on MSA benchmarks

Provides competitive performance on MER tasks

Reduces parameter and computational costs

Abstract

Multi-task learning (MTL) enables the efficient transfer of extra knowledge acquired from other tasks. The high correlation between multimodal sentiment analysis (MSA) and multimodal emotion recognition (MER) supports their joint training. However, existing methods primarily employ hard parameter sharing, ignoring parameter conflicts caused by complex task correlations. In this paper, we present a novel MTL method for MSA and MER, termed Multimodal Mixture of Low-Rank Experts (MMoLRE). MMoLRE utilizes shared and task-specific experts to distinctly model common and unique task characteristics, thereby avoiding parameter conflicts. Additionally, inspired by low-rank structures in the Mixture of Experts (MoE) framework, we design low-rank expert networks to reduce parameter and computational overhead as the number of experts increases. Extensive experiments on the CMU-MOSI and CMU-MOSEI benchmarks demonstrate that MMoLRE achieves state-of-the-art performance on the MSA task and competitive results on the MER task.