Towards Unified Modeling in Federated Multi-Task Learning via Subspace Decoupling

TL;DR

This paper introduces FedDEA, a novel federated multi-task learning method that decouples task-specific updates to improve joint training across heterogeneous tasks without requiring task labels or architecture changes.

Contribution

FedDEA provides a task-relevant dimension identification and rescaling mechanism for effective multi-task model aggregation in federated learning, addressing heterogeneity challenges.

Findings

Significant performance improvements on NYUD-V2 and PASCAL-Context datasets.

Robustness and generalization under highly heterogeneous task settings.

Compatible with various federated optimization algorithms.

Abstract

Federated Multi-Task Learning (FMTL) enables multiple clients performing heterogeneous tasks without exchanging their local data, offering broad potential for privacy preserving multi-task collaboration. However, most existing methods focus on building personalized models for each client and unable to support the aggregation of multiple heterogeneous tasks into a unified model. As a result, in real-world scenarios where task objectives, label spaces, and optimization paths vary significantly, conventional FMTL methods struggle to achieve effective joint training. To address this challenge, we propose FedDEA (Federated Decoupled Aggregation), an update-structure-aware aggregation method specifically designed for multi-task model integration. Our method dynamically identifies task-relevant dimensions based on the response strength of local updates and enhances their optimization effectiveness through rescaling. This mechanism effectively suppresses cross-task interference and enables task-level decoupled aggregation within a unified global model. FedDEA does not rely on task labels or architectural modifications, making it broadly applicable and deployment-friendly. Experimental results demonstrate that it can be easily integrated into various mainstream federated optimization algorithms and consistently delivers significant overall performance improvements on widely used NYUD-V2 and PASCAL-Context. These results validate the robustness and generalization capabilities of FedDEA under highly heterogeneous task settings.