Many-Task Federated Fine-Tuning via Unified Task Vectors

TL;DR

This paper introduces MaTU, a novel federated learning method that learns unified task vectors across clients, enabling efficient multi-task collaboration without client clustering or individual model management, and achieves superior results with reduced communication.

Contribution

MaTU is the first approach to jointly learn task vectors in federated settings, eliminating the need for clustering and client-specific models, and incorporates lightweight modulators for task-specific adaptation.

Findings

MaTU outperforms existing MaT-FL methods on 30 datasets.

Achieves results comparable to per-task fine-tuning.

Reduces communication costs significantly.

Abstract

Federated Learning (FL) traditionally assumes homogeneous client tasks; however, in real-world scenarios, clients often specialize in diverse tasks, introducing task heterogeneity. To address this challenge, Many-Task FL (MaT-FL) has emerged, enabling clients to collaborate effectively despite task diversity. Existing MaT-FL approaches rely on client grouping or personalized layers, requiring the server to manage individual models and failing to account for clients handling multiple tasks. We propose MaTU, a MaT-FL approach that enables joint learning of task vectors across clients, eliminating the need for clustering or client-specific weight storage at the server. Our method introduces a novel aggregation mechanism that determines task similarity based on the direction of clients task vectors and constructs a unified task vector encapsulating all tasks. To address task-specific requirements, we augment the unified task vector with lightweight modulators that facilitate knowledge transfer among related tasks while disentangling dissimilar ones. Evaluated across 30 datasets, MaTU achieves superior performance over state-of-the-art MaT-FL approaches, with results comparable to per-task fine-tuning, while delivering significant communication savings.