Multimodal Federated Learning on IoT Data

TL;DR

This paper introduces a multimodal federated learning framework that leverages multiple data modalities on IoT devices to enhance classification accuracy while preserving privacy.

Contribution

It proposes a novel semi-supervised multimodal federated learning approach using autoencoders and a new aggregation algorithm for improved performance across diverse data types.

Findings

Multimodal data improves federated learning classification accuracy.

The framework achieves over 60% F1 score using only one modality for supervision.

Combining unimodal and multimodal clients enhances overall performance.

Abstract

Federated learning is proposed as an alternative to centralized machine learning since its client-server structure provides better privacy protection and scalability in real-world applications. In many applications, such as smart homes with Internet-of-Things (IoT) devices, local data on clients are generated from different modalities such as sensory, visual, and audio data. Existing federated learning systems only work on local data from a single modality, which limits the scalability of the systems. In this paper, we propose a multimodal and semi-supervised federated learning framework that trains autoencoders to extract shared or correlated representations from different local data modalities on clients. In addition, we propose a multimodal FedAvg algorithm to aggregate local autoencoders trained on different data modalities. We use the learned global autoencoder for a downstream classification task with the help of auxiliary labelled data on the server. We empirically evaluate our framework on different modalities including sensory data, depth camera videos, and RGB camera videos. Our experimental results demonstrate that introducing data from multiple modalities into federated learning can improve its classification performance. In addition, we can use labelled data from only one modality for supervised learning on the server and apply the learned model to testing data from other modalities to achieve decent F1 scores (e.g., with the best performance being higher than 60%), especially when combining contributions from both unimodal clients and multimodal clients.