Modality Alignment Meets Federated Broadcasting

TL;DR

This paper proposes a federated learning framework using modality alignment inspired by multi-modal models like CLIP, which improves model performance and robustness across heterogeneous data distributions by aligning cross-client learning through a server-client communication scheme.

Contribution

It introduces a novel FL approach with modality alignment, leveraging pre-trained models and low-rank adaptation to enhance performance in heterogeneous data scenarios.

Findings

Improves model generalization in heterogeneous FL settings

Maintains robustness across diverse data distributions

Enhances communication efficiency with parameter updates

Abstract

Federated learning (FL) has emerged as a powerful approach to safeguard data privacy by training models across distributed edge devices without centralizing local data. Despite advancements in homogeneous data scenarios, maintaining performance between the global and local clients in FL over heterogeneous data remains challenging due to data distribution variations that degrade model convergence and increase computational costs. This paper introduces a novel FL framework leveraging modality alignment, where a text encoder resides on the server, and image encoders operate on local devices. Inspired by multi-modal learning paradigms like CLIP, this design aligns cross-client learning by treating server-client communications akin to multi-modal broadcasting. We initialize with a pre-trained model to mitigate overfitting, updating select parameters through low-rank adaptation (LoRA) to meet computational demand and performance efficiency. Local models train independently and communicate updates to the server, which aggregates parameters via a query-based method, facilitating cross-client knowledge sharing and performance improvement under extreme heterogeneity. Extensive experiments on benchmark datasets demonstrate the efficacy in maintaining generalization and robustness, even in highly heterogeneous settings.