MMBind: Unleashing the Potential of Distributed and Heterogeneous Data for Multimodal Learning in IoT

TL;DR

MMBind introduces a novel method for multimodal learning in IoT that constructs pseudo-paired datasets and employs weighted contrastive learning to effectively utilize distributed, heterogeneous, and incomplete data sources.

Contribution

It proposes a new data binding and learning framework that enables multimodal models to be trained on distributed, heterogeneous, and incomplete IoT data.

Findings

Outperforms state-of-the-art methods on ten real-world datasets

Effectively handles data incompleteness and domain shifts

Enhances multimodal foundation model training in IoT

Abstract

Multimodal sensing systems are increasingly prevalent in various real-world applications. Most existing multimodal learning approaches heavily rely on training with a large amount of synchronized, complete multimodal data. However, such a setting is impractical in real-world IoT sensing applications where data is typically collected by distributed nodes with heterogeneous data modalities, and is also rarely labeled. In this paper, we propose MMBind, a new data binding approach for multimodal learning on distributed and heterogeneous IoT data. The key idea of MMBind is to construct a pseudo-paired multimodal dataset for model training by binding data from disparate sources and incomplete modalities through a sufficiently descriptive shared modality. We also propose a weighted contrastive learning approach to handle domain shifts among disparate data, coupled with an adaptive multimodal learning architecture capable of training models with heterogeneous modality combinations. Evaluations on ten real-world multimodal datasets highlight that MMBind outperforms state-of-the-art baselines under varying degrees of data incompleteness and domain shift, and holds promise for advancing multimodal foundation model training in IoT applications\footnote (The source code is available via https://github.com/nesl/multimodal-bind).