Federated Learning from Small Datasets

TL;DR

This paper introduces a novel federated learning method that combines model aggregation with permutations of local models, enabling effective training on extremely small datasets while maintaining privacy and efficiency.

Contribution

The paper proposes a new approach that interweaves model aggregation with model permutations to improve federated learning on small datasets.

Findings

Enables training on very small local datasets such as patient data across hospitals.

Retains training efficiency and privacy benefits of federated learning.

Improves convergence in data-sparse domains.

Abstract

Federated learning allows multiple parties to collaboratively train a joint model without sharing local data. This enables applications of machine learning in settings of inherently distributed, undisclosable data such as in the medical domain. In practice, joint training is usually achieved by aggregating local models, for which local training objectives have to be in expectation similar to the joint (global) objective. Often, however, local datasets are so small that local objectives differ greatly from the global objective, resulting in federated learning to fail. We propose a novel approach that intertwines model aggregations with permutations of local models. The permutations expose each local model to a daisy chain of local datasets resulting in more efficient training in data-sparse domains. This enables training on extremely small local datasets, such as patient data across hospitals, while retaining the training efficiency and privacy benefits of federated learning.