Harnessing Increased Client Participation with Cohort-Parallel Federated Learning

TL;DR

This paper introduces Cohort-Parallel Federated Learning (CPFL), a novel approach that partitions clients into cohorts to improve training efficiency and resource usage while maintaining accuracy.

Contribution

The paper proposes CPFL, a new federated learning method that trains smaller cohorts independently and unifies models via knowledge distillation, enhancing efficiency over traditional FL.

Findings

CPFL with four cohorts reduces training time by 1.9x.

CPFL decreases resource usage by 1.3x.

Minimal accuracy drop observed with CPFL on CIFAR-10.

Abstract

Federated learning (FL) is a machine learning approach where nodes collaboratively train a global model. As more nodes participate in a round of FL, the effectiveness of individual model updates by nodes also diminishes. In this study, we increase the effectiveness of client updates by dividing the network into smaller partitions, or cohorts. We introduce Cohort-Parallel Federated Learning (CPFL): a novel learning approach where each cohort independently trains a global model using FL, until convergence, and the produced models by each cohort are then unified using knowledge distillation. The insight behind CPFL is that smaller, isolated networks converge quicker than in a one-network setting where all nodes participate. Through exhaustive experiments involving realistic traces and non-IID data distributions on the CIFAR-10 and FEMNIST image classification tasks, we investigate the balance between the number of cohorts, model accuracy, training time, and compute resources. Compared to traditional FL, CPFL with four cohorts, non-IID data distribution, and CIFAR-10 yields a 1.9x reduction in train time and a 1.3x reduction in resource usage, with a minimal drop in test accuracy.