Single-shot Hyper-parameter Optimization for Federated Learning: A General Algorithm & Analysis

TL;DR

This paper introduces FLoRA, a novel federated learning hyper-parameter optimization framework that efficiently finds a single optimal hyper-parameter set with minimal communication, improving model accuracy across diverse datasets.

Contribution

FLoRA is a general, single-shot FL-HPO algorithm applicable to various ML models and data types, with theoretical analysis of its optimality gap considering data heterogeneity.

Findings

FLoRA achieves significant accuracy improvements over baselines.

FLoRA maintains robustness as the number of parties increases.

Theoretical analysis accounts for non-IID data heterogeneity.

Abstract

We address the relatively unexplored problem of hyper-parameter optimization (HPO) for federated learning (FL-HPO). We introduce Federated Loss SuRface Aggregation (FLoRA), a general FL-HPO solution framework that can address use cases of tabular data and any Machine Learning (ML) model including gradient boosting training algorithms and therefore further expands the scope of FL-HPO. FLoRA enables single-shot FL-HPO: identifying a single set of good hyper-parameters that are subsequently used in a single FL training. Thus, it enables FL-HPO solutions with minimal additional communication overhead compared to FL training without HPO. We theoretically characterize the optimality gap of FL-HPO, which explicitly accounts for the heterogeneous non-IID nature of the parties' local data distributions, a dominant characteristic of FL systems. Our empirical evaluation of FLoRA for multiple ML algorithms on seven OpenML datasets demonstrates significant model accuracy improvements over the considered baseline, and robustness to increasing number of parties involved in FL-HPO training.