Multimodal Online Federated Learning with Modality Missing in Internet of Things

TL;DR

This paper introduces MMO-FL, a novel online federated learning framework for multimodal IoT data that effectively handles missing modalities using the PMM algorithm, with theoretical analysis and experimental validation.

Contribution

It proposes MMO-FL for dynamic decentralized multimodal learning in IoT and introduces PMM to address missing modality challenges.

Findings

PMM outperforms benchmark methods in experiments.

Theoretical analysis shows performance degradation due to missing modalities.

MMO-FL effectively manages multimodal data in real-time IoT environments.

Abstract

The Internet of Things (IoT) ecosystem generates vast amounts of multimodal data from heterogeneous sources such as sensors, cameras, and microphones. As edge intelligence continues to evolve, IoT devices have progressed from simple data collection units to nodes capable of executing complex computational tasks. This evolution necessitates the adoption of distributed learning strategies to effectively handle multimodal data in an IoT environment. Furthermore, the real-time nature of data collection and limited local storage on edge devices in IoT call for an online learning paradigm. To address these challenges, we introduce the concept of Multimodal Online Federated Learning (MMO-FL), a novel framework designed for dynamic and decentralized multimodal learning in IoT environments. Building on this framework, we further account for the inherent instability of edge devices, which frequently results in missing modalities during the learning process. We conduct a comprehensive theoretical analysis under both complete and missing modality scenarios, providing insights into the performance degradation caused by missing modalities. To mitigate the impact of modality missing, we propose the Prototypical Modality Mitigation (PMM) algorithm, which leverages prototype learning to effectively compensate for missing modalities. Experimental results on two multimodal datasets further demonstrate the superior performance of PMM compared to benchmarks.