Temporal-Spatial Decouple before Act: Disentangled Representation Learning for Multimodal Sentiment Analysis

TL;DR

This paper introduces TSDA, a novel approach for multimodal sentiment analysis that explicitly decouples temporal and spatial features of each modality before interaction, improving performance over existing methods.

Contribution

The paper proposes a new decoupling framework that separates temporal and spatial information in multimodal data, with cross-modal alignment and recoupling, enhancing analysis accuracy.

Findings

TSDA outperforms baseline models in multimodal sentiment analysis tasks.

Decoupling temporal and spatial features improves interpretability and effectiveness.

Ablation studies confirm the importance of each component in TSDA.

Abstract

Multimodal Sentiment Analysis integrates Linguistic, Visual, and Acoustic. Mainstream approaches based on modality-invariant and modality-specific factorization or on complex fusion still rely on spatiotemporal mixed modeling. This ignores spatiotemporal heterogeneity, leading to spatiotemporal information asymmetry and thus limited performance. Hence, we propose TSDA, Temporal-Spatial Decouple before Act, which explicitly decouples each modality into temporal dynamics and spatial structural context before any interaction. For every modality, a temporal encoder and a spatial encoder project signals into separate temporal and spatial body. Factor-Consistent Cross-Modal Alignment then aligns temporal features only with their temporal counterparts across modalities, and spatial features only with their spatial counterparts. Factor specific supervision and decorrelation regularization reduce cross factor leakage while preserving complementarity. A Gated Recouple module subsequently recouples the aligned streams for task. Extensive experiments show that TSDA outperforms baselines. Ablation analysis studies confirm the necessity and interpretability of the design.