Why Federated Optimization Fails to Achieve Perfect Fitting? A Theoretical Perspective on Client-Side Optima

TL;DR

This paper offers a theoretical explanation for why federated optimization struggles with perfect data fitting under data heterogeneity, showing that client data differences lead to local optima divergence and oscillations that limit model performance.

Contribution

It introduces a new theoretical framework explaining performance degradation in federated learning due to data heterogeneity and local optima divergence.

Findings

Local optima divergence increases the lower bound of the global objective.

Global model oscillates within a region instead of converging to a single optimum.

Theoretical insights are validated through experiments across multiple tasks.

Abstract

Federated optimization is a constrained form of distributed optimization that enables training a global model without directly sharing client data. Although existing algorithms can guarantee convergence in theory and often achieve stable training in practice, the reasons behind performance degradation under data heterogeneity remain unclear. To address this gap, the main contribution of this paper is to provide a theoretical perspective that explains why such degradation occurs. We introduce the assumption that heterogeneous client data lead to distinct local optima, and show that this assumption implies two key consequences: 1) the distance among clients' local optima raises the lower bound of the global objective, making perfect fitting of all client data impossible; and 2) in the final training stage, the global model oscillates within a region instead of converging to a single optimum, limiting its ability to fully fit the data. These results provide a principled explanation for performance degradation in non-iid settings, which we further validate through experiments across multiple tasks and neural network architectures. The framework used in this paper is open-sourced at: https://github.com/NPCLEI/fedtorch.