LEAF: A Benchmark for Federated Settings

TL;DR

LEAF is a comprehensive benchmarking framework designed to evaluate federated learning algorithms using diverse datasets, evaluation protocols, and reference implementations, addressing real-world challenges in federated environments.

Contribution

This paper introduces LEAF, the first modular benchmark suite for federated learning that includes datasets, evaluation tools, and reference models to facilitate realistic research.

Findings

Provides diverse federated datasets for benchmarking.

Establishes standardized evaluation protocols.

Includes reference implementations for reproducibility.

Abstract

Modern federated networks, such as those comprised of wearable devices, mobile phones, or autonomous vehicles, generate massive amounts of data each day. This wealth of data can help to learn models that can improve the user experience on each device. However, the scale and heterogeneity of federated data presents new challenges in research areas such as federated learning, meta-learning, and multi-task learning. As the machine learning community begins to tackle these challenges, we are at a critical time to ensure that developments made in these areas are grounded with realistic benchmarks. To this end, we propose LEAF, a modular benchmarking framework for learning in federated settings. LEAF includes a suite of open-source federated datasets, a rigorous evaluation framework, and a set of reference implementations, all geared towards capturing the obstacles and intricacies of practical federated environments.