ColNet: Collaborative Optimization in Decentralized Federated Multi-task Learning Systems

TL;DR

ColNet introduces a decentralized federated multi-task learning framework that effectively handles task heterogeneity through adaptive clustering and group-based model aggregation, outperforming existing methods in diverse scenarios.

Contribution

It proposes a novel decentralized FMTL framework, ColNet, with adaptive clustering and group aggregation to address task heterogeneity and improve robustness.

Findings

Outperforms competing schemes under label and task heterogeneity

Demonstrates robustness to poisoning attacks

Effective in decentralized federated environments

Abstract

The integration of Federated Learning (FL) and Multi-Task Learning (MTL) has been explored to address client heterogeneity, with Federated Multi-Task Learning (FMTL) treating each client as a distinct task. However, most existing research focuses on data heterogeneity (e.g., addressing non-IID data) rather than task heterogeneity, where clients solve fundamentally different tasks. Additionally, much of the work relies on centralized settings with a server managing the federation, leaving the more challenging domain of decentralized FMTL largely unexplored. Thus, this work bridges this gap by proposing ColNet, a framework designed for heterogeneous tasks in decentralized federated environments. ColNet partitions models into a backbone and task-specific heads, and uses adaptive clustering based on model and data sensitivity to form task-coherent client groups. Backbones are averaged within groups, and group leaders perform hyper-conflict-averse cross-group aggregation. Across datasets and federations, ColNet outperforms competing schemes under label and task heterogeneity and shows robustness to poisoning attacks.