A Hierarchical Sampling Framework for bounding the Generalization Error of Federated Learning

TL;DR

This paper develops a hierarchical sampling framework for federated learning, deriving Wasserstein-based generalization bounds that incorporate client dependencies and privacy considerations.

Contribution

It introduces a novel hierarchical sampling model and derives Wasserstein-based generalization bounds that unify and extend existing mutual information and privacy bounds.

Findings

Bounds recover the asymptotic rate of generalization error in the Gaussian Location Model.

Framework unifies existing mutual information and differential privacy bounds.

Demonstrates the tightness of bounds through theoretical analysis.

Abstract

We study expected generalization bounds for the Hierarchical Federated Learning (HFL) setup using Wasserstein distance. We introduce a generalized framework in which data is sampled hierarchically, and we model it with a multi-layered tree structure that induces dependencies among the clients' datasets. We derive generalization bounds in terms of Wasserstein distance under the Lipschitz assumption on the loss function, by applying a supersample construction that allows us to measure the sensitivity of the algorithm to the change of a single node in the sampling tree. By leveraging the FL structure, we recover and strictly imply existing state-of-the-art conditional mutual information (CMI) bounds in the case of bounded losses. We also show that our bound can be applied together with Differential Privacy assumptions, to recover generalization bounds based on algorithmic privacy. To assess the tightness of our bounds, we study the Gaussian Location Model (GLM) and show that we recover the actual asymptotic rate of the generalization error.