Can We Theoretically Quantify the Impacts of Local Updates on the Generalization Performance of Federated Learning?

TL;DR

This paper provides a theoretical analysis of how local updates and data heterogeneity affect the generalization performance of federated learning, offering closed-form expressions and insights into model behavior over time.

Contribution

It introduces a comprehensive theoretical framework quantifying the impact of local update frequency and data heterogeneity on FL's generalization, using a linear model analysis.

Findings

Quantifies the effect of local updates on generalization error.

Provides closed-form expressions for model error in FL.

Offers insights into benign overfitting in federated learning.

Abstract

Federated Learning (FL) has gained significant popularity due to its effectiveness in training machine learning models across diverse sites without requiring direct data sharing. While various algorithms along with their optimization analyses have shown that FL with local updates is a communication-efficient distributed learning framework, the generalization performance of FL with local updates has received comparatively less attention. This lack of investigation can be attributed to the complex interplay between data heterogeneity and infrequent communication due to the local updates within the FL framework. This motivates us to investigate a fundamental question in FL: Can we quantify the impact of data heterogeneity and local updates on the generalization performance for FL as the learning process evolves? To this end, we conduct a comprehensive theoretical study of FL's generalization performance using a linear model as the first step, where the data heterogeneity is considered for both the stationary and online/non-stationary cases. By providing closed-form expressions of the model error, we rigorously quantify the impact of the number of the local updates (denoted as $K$) under three settings ($K=1$, $K<\infty$, and $K=\infty$) and show how the generalization performance evolves with the number of rounds $t$. Our investigation also provides a comprehensive understanding of how different configurations (including the number of model parameters $p$ and the number of training samples $n$) contribute to the overall generalization performance, thus shedding new insights (such as benign overfitting) for implementing FL over networks.