Collaborative Batch Size Optimization for Federated Learning

TL;DR

This paper proposes a greedy randomized search method to optimize local batch sizes in federated learning, enhancing convergence speed without requiring inter-participant communication.

Contribution

It introduces a novel hardware-aware batch size optimization technique for federated learning that improves training efficiency.

Findings

Improved convergence speed over default settings

Nearly matches performance of fully optimized local parameters

Effective in heterogeneous hardware environments

Abstract

Federated Learning (FL) is a decentralized collaborative Machine Learning framework for training models without collecting data in a centralized location. It has seen application across various disciplines, from helping medical diagnoses in hospitals to detecting fraud in financial transactions. In this paper, we focus on improving the local training process through hardware usage optimization. While participants in a federation might share the hardware they are training on, since there is no information exchange between them, their training process can be hindered by an improper training configuration. Taking advantage of the parallel processing inherent to Federated Learning, we use a greedy randomized search to optimize local batch sizes for the best training settings across all participants. Our results show that against default parameter settings, our method improves convergence speed while staying nearly on par with the case where local parameters are optimized.