Text-Routed Sparse Mixture-of-Experts Model with Explanation and Temporal Alignment for Multi-Modal Sentiment Analysis

TL;DR

This paper introduces TEXT, a multi-modal sentiment analysis model that combines explanations from large language models with temporal alignment techniques, achieving superior performance across multiple datasets.

Contribution

The paper presents a novel text-routed sparse mixture-of-experts model with explanation and temporal alignment, enhancing multi-modal sentiment analysis performance.

Findings

TEXT outperforms recent models on four datasets

Achieves a 13.5% reduction in mean absolute error on CH-SIMS

Wins on at least four metrics across six evaluated metrics

Abstract

Human-interaction-involved applications underscore the need for Multi-modal Sentiment Analysis (MSA). Although many approaches have been proposed to address the subtle emotions in different modalities, the power of explanations and temporal alignments is still underexplored. Thus, this paper proposes the Text-routed sparse mixture-of-Experts model with eXplanation and Temporal alignment for MSA (TEXT). TEXT first augments explanations for MSA via Multi-modal Large Language Models (MLLM), and then novelly aligns the epresentations of audio and video through a temporality-oriented neural network block. TEXT aligns different modalities with explanations and facilitates a new text-routed sparse mixture-of-experts with gate fusion. Our temporal alignment block merges the benefits of Mamba and temporal cross-attention. As a result, TEXT achieves the best performance cross four datasets among all tested models, including three recently proposed approaches and three MLLMs. TEXT wins on at least four metrics out of all six metrics. For example, TEXT decreases the mean absolute error to 0.353 on the CH-SIMS dataset, which signifies a 13.5% decrement compared with recently proposed approaches.