Federated Bayesian Neural Regression: A Scalable Global Federated Gaussian Process

TL;DR

This paper introduces FedBNR, a scalable federated Bayesian neural regression method that learns a global Gaussian process model while preserving client privacy, improving regression accuracy in non-i.i.d. settings.

Contribution

It proposes a novel federated GP framework using deep kernel learning and random features, enabling privacy-preserving, scalable, and accurate global modeling.

Findings

Significant performance improvements over existing federated GP models.

Effective learning of global kernels for non-i.i.d. client data.

Demonstrated scalability with real-world datasets.

Abstract

In typical scenarios where the Federated Learning (FL) framework applies, it is common for clients to have insufficient training data to produce an accurate model. Thus, models that provide not only point estimations, but also some notion of confidence are beneficial. Gaussian Process (GP) is a powerful Bayesian model that comes with naturally well-calibrated variance estimations. However, it is challenging to learn a stand-alone global GP since merging local kernels leads to privacy leakage. To preserve privacy, previous works that consider federated GPs avoid learning a global model by focusing on the personalized setting or learning an ensemble of local models. We present Federated Bayesian Neural Regression (FedBNR), an algorithm that learns a scalable stand-alone global federated GP that respects clients' privacy. We incorporate deep kernel learning and random features for scalability by defining a unifying random kernel. We show this random kernel can recover any stationary kernel and many non-stationary kernels. We then derive a principled approach of learning a global predictive model as if all client data is centralized. We also learn global kernels with knowledge distillation methods for non-identically and independently distributed (non-i.i.d.) clients. Experiments are conducted on real-world regression datasets and show statistically significant improvements compared to other federated GP models.