StreaMulT: Streaming Multimodal Transformer for Heterogeneous and Arbitrary Long Sequential Data

TL;DR

StreaMulT introduces a novel streaming multimodal transformer model designed to handle arbitrarily long, heterogeneous data streams for predictive maintenance, outperforming existing models on sentiment analysis benchmarks.

Contribution

This paper formalizes the streaming multimodal learning paradigm and proposes StreaMulT, a transformer-based model that processes long, unaligned multimodal data streams using cross-modal attention and memory.

Findings

Outperforms state-of-the-art on CMU-MOSEI sentiment analysis dataset

Handles arbitrarily long multimodal input sequences

Highlights the significance of textual embeddings in multimodal tasks

Abstract

The increasing complexity of Industry 4.0 systems brings new challenges regarding predictive maintenance tasks such as fault detection and diagnosis. A corresponding and realistic setting includes multi-source data streams from different modalities, such as sensors measurements time series, machine images, textual maintenance reports, etc. These heterogeneous multimodal streams also differ in their acquisition frequency, may embed temporally unaligned information and can be arbitrarily long, depending on the considered system and task. Whereas multimodal fusion has been largely studied in a static setting, to the best of our knowledge, there exists no previous work considering arbitrarily long multimodal streams alongside with related tasks such as prediction across time. Thus, in this paper, we first formalize this paradigm of heterogeneous multimodal learning in a streaming setting as a new one. To tackle this challenge, we propose StreaMulT, a Streaming Multimodal Transformer relying on cross-modal attention and on a memory bank to process arbitrarily long input sequences at training time and run in a streaming way at inference. StreaMulT improves the state-of-the-art metrics on CMU-MOSEI dataset for Multimodal Sentiment Analysis task, while being able to deal with much longer inputs than other multimodal models. The conducted experiments eventually highlight the importance of the textual embedding layer, questioning recent improvements in Multimodal Sentiment Analysis benchmarks.