Pollen: High-throughput Federated Learning Simulation via Resource-Aware Client Placement

TL;DR

Pollen is a resource-aware system that significantly accelerates large-scale federated learning simulations by optimizing client placement and scheduling, addressing communication inefficiencies and hardware heterogeneity.

Contribution

Pollen introduces a novel push-based client placement and adaptive scheduling approach to improve simulation speed and resource utilization in federated learning.

Findings

Pollen increases GPU utilization and reduces idle time.

Experimental speed-ups of days or weeks over existing simulators.

Effective handling of heterogeneous hardware in FL simulations.

Abstract

Federated Learning (FL) is a privacy-focused machine learning paradigm that collaboratively trains models directly on edge devices. Simulation plays an essential role in FL adoption, helping develop novel aggregation and client sampling strategies. However, current simulators cannot emulate large-scale systems in a time-efficient manner, which limits their utility and casts doubts on generalizability. This work proposes Pollen, a novel resource-aware system for speeding up simulations. Pollen addresses two limiting factors from existing simulators: (a) communication inefficiency derived from pull-based client execution and (b) inadequate load balance when using heterogeneous hardware. Pollen executes high-throughput FL simulations at scale by (a) using a push-based client placement system, (b) learning how an adaptable scheduling of clients based on hardware statistics (c) estimating the optimal number of concurrent workers per GPU. We evaluate Pollen on four representative FL tasks and show that Pollen's placement model increases GPU utilization and reduces idle time. We compare Pollen to Flower, Flute, FedScale, Parrot, and pfl and show experimental speed-ups of days or weeks.