Factorized Multimodal Transformer for Multimodal Sequential Learning

TL;DR

The paper introduces the Factorized Multimodal Transformer (FMT), a novel model designed to effectively learn complex multimodal sequential data by modeling intra- and intermodal dynamics with enhanced attention mechanisms, achieving state-of-the-art results.

Contribution

The paper proposes a new transformer architecture, FMT, that factorizes multimodal dynamics, enabling better modeling of complex data without overfitting, even with limited resources.

Findings

FMT outperforms previous models on multiple datasets

Achieves new state-of-the-art performance in multimodal tasks

Effectively models long-range multimodal dependencies

Abstract

The complex world around us is inherently multimodal and sequential (continuous). Information is scattered across different modalities and requires multiple continuous sensors to be captured. As machine learning leaps towards better generalization to real world, multimodal sequential learning becomes a fundamental research area. Arguably, modeling arbitrarily distributed spatio-temporal dynamics within and across modalities is the biggest challenge in this research area. In this paper, we present a new transformer model, called the Factorized Multimodal Transformer (FMT) for multimodal sequential learning. FMT inherently models the intramodal and intermodal (involving two or more modalities) dynamics within its multimodal input in a factorized manner. The proposed factorization allows for increasing the number of self-attentions to better model the multimodal phenomena at hand; without encountering difficulties during training (e.g. overfitting) even on relatively low-resource setups. All the attention mechanisms within FMT have a full time-domain receptive field which allows them to asynchronously capture long-range multimodal dynamics. In our experiments we focus on datasets that contain the three commonly studied modalities of language, vision and acoustic. We perform a wide range of experiments, spanning across 3 well-studied datasets and 21 distinct labels. FMT shows superior performance over previously proposed models, setting new state of the art in the studied datasets.