Function-Space ADMM for Decentralized Federated Learning: A Control Theoretic Perspective

TL;DR

This paper introduces FedF-ADMM, a novel decentralized federated learning method leveraging function space optimization and control theory, achieving faster, more stable convergence in non-IID data scenarios.

Contribution

It proposes a function-space ADMM approach for decentralized FL, incorporating a stabilization coefficient analyzed via control theory, improving convergence and robustness over existing methods.

Findings

FedF-ADMM converges faster than existing methods.

It maintains higher accuracy in non-IID data settings.

The stabilization coefficient enhances robustness under severe non-IID conditions.

Abstract

Decentralized federated learning (FL) is a promising approach for training machine learning models on sensor networks, Internet of Things (IoT) devices, and other edge systems where no central server exists. While federated learning offers advantages such as preserving data privacy, it often suffers from non-independent and identically distributed (IID) data distributions across devices, which cause significant performance degradation. This issue is particularly severe when directly optimizing model parameters, because neural network training is inherently non-convex and standard convergence guarantees for convex optimization do not apply. Unlike existing decentralized FL methods that primarily operate in parameter space, we propose federated function-space alternating direction method of multipliers (FedF-ADMM). FedF-ADMM exploits the convexity of loss functionals within function space to derive alternating direction method of multipliers (ADMM)-based update directions, which are subsequently projected onto the parameter space via knowledge distillation. We further introduce a stabilization coefficient to enhance robustness under severe non-IID settings and analyze its behavior from a control-theoretic perspective by interpreting it as a proportional-integral (PI) term. Experiments under challenging non-IID scenarios, including settings where each device has data from only a single label, demonstrate that FedF-ADMM achieves faster and more stable convergence than existing decentralized FL methods, while attaining higher accuracy and better consensus among devices.