A Self-Adjusting Fusion Representation Learning Model for Unaligned Text-Audio Sequences

TL;DR

This paper introduces a novel self-adjusting fusion model for unaligned text-audio sequences in multimodal sentiment analysis, effectively capturing inter-modal interactions while preserving unimodal features.

Contribution

The proposed SA-FRLM model uniquely combines crossmodal alignment, collaboration attention, and a crossmodal adjustment transformer to improve fusion representation learning from unaligned modalities.

Findings

Significant performance improvements on CMU-MOSI and CMU-MOSEI datasets.

Effective modeling of unaligned text-audio sequences.

Enhanced preservation of unimodal characteristics.

Abstract

Inter-modal interaction plays an indispensable role in multimodal sentiment analysis. Due to different modalities sequences are usually non-alignment, how to integrate relevant information of each modality to learn fusion representations has been one of the central challenges in multimodal learning. In this paper, a Self-Adjusting Fusion Representation Learning Model (SA-FRLM) is proposed to learn robust crossmodal fusion representations directly from the unaligned text and audio sequences. Different from previous works, our model not only makes full use of the interaction between different modalities but also maximizes the protection of the unimodal characteristics. Specifically, we first employ a crossmodal alignment module to project different modalities features to the same dimension. The crossmodal collaboration attention is then adopted to model the inter-modal interaction between text and audio sequences and initialize the fusion representations. After that, as the core unit of the SA-FRLM, the crossmodal adjustment transformer is proposed to protect original unimodal characteristics. It can dynamically adapt the fusion representations by using single modal streams. We evaluate our approach on the public multimodal sentiment analysis datasets CMU-MOSI and CMU-MOSEI. The experiment results show that our model has significantly improved the performance of all the metrics on the unaligned text-audio sequences.