Communication-Efficient Personalized Adaptation via Federated-Local Model Merging

TL;DR

Potara is a federated learning framework that efficiently merges a general federated model with personalized local models, improving task-specific adaptation while reducing communication costs.

Contribution

It introduces a theoretically grounded model merging approach with closed-form optimal weights, enhancing personalization in federated settings.

Findings

Achieves better personalization performance on vision and language tasks.

Reduces communication overhead compared to existing methods.

Provides theoretical guarantees for model merging effectiveness.

Abstract

Parameter-efficient fine-tuning methods, such as LoRA, offer a practical way to adapt large vision and language models to client tasks. However, this becomes particularly challenging under task-level heterogeneity in federated deployments. In this regime, personalization requires balancing general knowledge with personalized knowledge, yet existing approaches largely rely on heuristic mixing rules and lack theoretical justification. Moreover, prior model merging approaches are also computation and communication intensive, making the process inefficient in federated settings. In this work, we propose Potara, a principled framework for federated personalization that constructs a personalized model for each client by merging two complementary models: (i) a federated model capturing general knowledge, and (ii) a local model capturing personalized knowledge. Through the construct of linear mode connectivity, we show that the expected task loss admits a variance trace upper bound, whose minimization yields closed-form optimal mixing weights that guarantee a tighter bound for the merged model than for either the federated or local model alone. Experiments on vision and language benchmarks show that Potara consistently improves personalization while reducing communication, leading to a strong performance-communication trade-off.