Federated Multi-Task Learning on Non-IID Data Silos: An Experimental Study

TL;DR

This paper introduces FMTL-Bench, a comprehensive evaluation framework for Federated Multi-Task Learning on Non-IID data, providing systematic comparisons and insights into various methods' performance and resource consumption.

Contribution

The paper presents a novel benchmark framework, FMTL-Bench, for evaluating FMTL methods across data, model, and optimization aspects with diverse Non-IID scenarios.

Findings

Benchmark reveals strengths and limitations of existing FMTL methods.

Systematic comparison of algorithms under various Non-IID data partitions.

Insights into communication, time, and energy efficiency of FMTL approaches.

Abstract

The innovative Federated Multi-Task Learning (FMTL) approach consolidates the benefits of Federated Learning (FL) and Multi-Task Learning (MTL), enabling collaborative model training on multi-task learning datasets. However, a comprehensive evaluation method, integrating the unique features of both FL and MTL, is currently absent in the field. This paper fills this void by introducing a novel framework, FMTL-Bench, for systematic evaluation of the FMTL paradigm. This benchmark covers various aspects at the data, model, and optimization algorithm levels, and comprises seven sets of comparative experiments, encapsulating a wide array of non-independent and identically distributed (Non-IID) data partitioning scenarios. We propose a systematic process for comparing baselines of diverse indicators and conduct a case study on communication expenditure, time, and energy consumption. Through our exhaustive experiments, we aim to provide valuable insights into the strengths and limitations of existing baseline methods, contributing to the ongoing discourse on optimal FMTL application in practical scenarios. The source code can be found on https://github.com/youngfish42/FMTL-Benchmark .