FedGradNorm: Personalized Federated Gradient-Normalized Multi-Task Learning

TL;DR

FedGradNorm is a novel method for personalized federated multi-task learning that normalizes gradient norms to balance learning speeds across tasks, improving convergence and handling data heterogeneity.

Contribution

Introduces FedGradNorm, a dynamic-weighting technique that enhances federated multi-task learning by balancing task training speeds and improving convergence.

Findings

Faster training convergence compared to equal-weighting strategies.

Effectively handles data and task heterogeneity in federated settings.

Improves personalized learning performance across diverse datasets.

Abstract

Multi-task learning (MTL) is a novel framework to learn several tasks simultaneously with a single shared network where each task has its distinct personalized header network for fine-tuning. MTL can be implemented in federated learning settings as well, in which tasks are distributed across clients. In federated settings, the statistical heterogeneity due to different task complexities and data heterogeneity due to non-iid nature of local datasets can both degrade the learning performance of the system. In addition, tasks can negatively affect each other's learning performance due to negative transference effects. To cope with these challenges, we propose FedGradNorm which uses a dynamic-weighting method to normalize gradient norms in order to balance learning speeds among different tasks. FedGradNorm improves the overall learning performance in a personalized federated learning setting. We provide convergence analysis for FedGradNorm by showing that it has an exponential convergence rate. We also conduct experiments on multi-task facial landmark (MTFL) and wireless communication system dataset (RadComDynamic). The experimental results show that our framework can achieve faster training performance compared to equal-weighting strategy. In addition to improving training speed, FedGradNorm also compensates for the imbalanced datasets among clients.