Just-in-Time Aggregation for Federated Learning

TL;DR

This paper introduces a 'just-in-time' aggregation method for federated learning that reduces resource consumption by delaying aggregation until necessary, without impacting job latency.

Contribution

The paper proposes a novel JIT aggregation paradigm for FL that leverages update periodicity to optimize resource usage and demonstrates significant efficiency gains.

Findings

Resource usage reduced by over 60% with JIT aggregation.

JIT aggregation introduces negligible overhead.

Effective across multiple datasets, models, and algorithms.

Abstract

The increasing number and scale of federated learning (FL) jobs necessitates resource efficient scheduling and management of aggregation to make the economics of cloud-hosted aggregation work. Existing FL research has focused on the design of FL algorithms and optimization, and less on the efficacy of aggregation. Existing FL platforms often employ aggregators that actively wait for model updates. This wastes computational resources on the cloud, especially in large scale FL settings where parties are intermittently available for training. In this paper, we propose a new FL aggregation paradigm -- "just-in-time" (JIT) aggregation that leverages unique properties of FL jobs, especially the periodicity of model updates, to defer aggregation as much as possible and free compute resources for other FL jobs or other datacenter workloads. We describe a novel way to prioritize FL jobs for aggregation, and demonstrate using multiple datasets, models and FL aggregation algorithms that our techniques can reduce resource usage by 60+\% when compared to eager aggregation used in existing FL platforms. We also demonstrate that using JIT aggregation has negligible overhead and impact on the latency of the FL job.