Co-LoRA: Collaborative Model Personalization on Heterogeneous Multi-Modal Clients

TL;DR

Co-LoRA advances personalized federated learning by enabling knowledge sharing across heterogeneous models and data, addressing real-world scenario complexities with a new benchmark and demonstrating superior performance.

Contribution

The paper introduces Co-LoRA, a novel dimension-invariant module for model aggregation in heterogeneous PFL, and a multi-modal benchmark with diverse tasks and distribution shifts.

Findings

Co-LoRA significantly outperforms existing PFL methods in heterogeneous settings.

The proposed task-relevance-aware aggregation reduces parameter interference.

The multi-modal benchmark effectively simulates real-world task diversity.

Abstract

As AI becomes more personal, e.g., Agentic AI, there is an increasing need for personalizing models for various use cases. Personalized federated learning (PFL) enables each client to collaboratively leverage other clients' knowledge for better adaptation to the task of interest, without privacy risks. Despite its potential, existing PFL methods remain confined to rather simplified scenarios where data and models are the same across clients. To move towards realistic scenarios, we move beyond these restrictive assumptions by addressing both data and model heterogeneity. We propose a task-relevance-aware model aggregation strategy to reduce parameter interference under heterogeneous data. Moreover, we introduce Co-LoRA, a dimension-invariant module that enables knowledge sharing across heterogeneous architectures. To mimic the real-world task diversity, we propose a multi-modal PFL benchmark spanning 40 distinct tasks with distribution shifts over time. Extensive experiments shows that our proposed method significantly outperforms the state-of-the-art PFL methods under heterogeneous scenarios.