Algorithms for Collaborative Machine Learning under Statistical Heterogeneity

TL;DR

This paper introduces three novel algorithms—SuPerFed, AAggFF, and FedEvg—that address statistical heterogeneity challenges in federated learning, enhancing personalization, performance consistency, and synthetic data generation for distributed machine learning.

Contribution

The paper presents new methods tailored to mitigate statistical heterogeneity in federated learning, improving personalization, client performance uniformity, and synthetic data generation.

Findings

SuPerFed improves personalization in heterogeneous data

AAggFF achieves uniform client performance through adaptive decision-making

FedEvg generates high-quality synthetic data for collaborative learning

Abstract

Learning from distributed data without accessing them is undoubtedly a challenging and non-trivial task. Nevertheless, the necessity for distributed training of a statistical model has been increasing, due to the privacy concerns of local data owners and the cost in centralizing the massively distributed data. Federated learning (FL) is currently the de facto standard of training a machine learning model across heterogeneous data owners, without leaving the raw data out of local silos. Nevertheless, several challenges must be addressed in order for FL to be more practical in reality. Among these challenges, the statistical heterogeneity problem is the most significant and requires immediate attention. From the main objective of FL, three major factors can be considered as starting points -- \textit{parameter}, textit{mixing coefficient}, and \textit{local data distributions}. In alignment with the components, this dissertation is organized into three parts. In Chapter II, a novel personalization method, \texttt{SuPerFed}, inspired by the mode-connectivity is introduced. In Chapter III, an adaptive decision-making algorithm, \texttt{AAggFF}, is introduced for inducing uniform performance distributions in participating clients, which is realized by online convex optimization framework. Finally, in Chapter IV, a collaborative synthetic data generation method, \texttt{FedEvg}, is introduced, leveraging the flexibility and compositionality of an energy-based modeling approach. Taken together, all of these approaches provide practical solutions to mitigate the statistical heterogeneity problem in data-decentralized settings, paving the way for distributed systems and applications using collaborative machine learning methods.