Federated PAC-Bayesian Learning on Non-IID data

TL;DR

This paper develops a novel federated PAC-Bayesian bound specifically designed for non-IID data, incorporating client-specific priors and adaptive weights, validated through experiments on real datasets.

Contribution

It introduces the first non-vacuous federated PAC-Bayesian bound for non-IID data, along with an optimization algorithm and objective function.

Findings

Bound tailored for non-IID data with client-specific priors

Proposed Gibbs-based algorithm for bound optimization

Validated on real-world datasets

Abstract

Existing research has either adapted the Probably Approximately Correct (PAC) Bayesian framework for federated learning (FL) or used information-theoretic PAC-Bayesian bounds while introducing their theorems, but few considering the non-IID challenges in FL. Our work presents the first non-vacuous federated PAC-Bayesian bound tailored for non-IID local data. This bound assumes unique prior knowledge for each client and variable aggregation weights. We also introduce an objective function and an innovative Gibbs-based algorithm for the optimization of the derived bound. The results are validated on real-world datasets.