Mode Connectivity and Data Heterogeneity of Federated Learning

TL;DR

This paper investigates the relationship between client and global modes in federated learning, revealing how data heterogeneity affects mode connectivity and proposing theoretical bounds to improve understanding and performance.

Contribution

It provides the first empirical and theoretical analysis of mode connectivity in federated learning, linking data heterogeneity to mode overlaps and establishing bounds on connectivity.

Findings

Reducing data heterogeneity improves mode connectivity.

Non-linear connectivity eliminates barriers present in linear paths.

Theoretical bounds relate connectivity to model width and data heterogeneity.

Abstract

Federated learning (FL) enables multiple clients to train a model while keeping their data private collaboratively. Previous studies have shown that data heterogeneity between clients leads to drifts across client updates. However, there are few studies on the relationship between client and global modes, making it unclear where these updates end up drifting. We perform empirical and theoretical studies on this relationship by utilizing mode connectivity, which measures performance change (i.e., connectivity) along parametric paths between different modes. Empirically, reducing data heterogeneity makes the connectivity on different paths more similar, forming more low-error overlaps between client and global modes. We also find that a barrier to connectivity occurs when linearly connecting two global modes, while it disappears with considering non-linear mode connectivity. Theoretically, we establish a quantitative bound on the global-mode connectivity using mean-field theory or dropout stability. The bound demonstrates that the connectivity improves when reducing data heterogeneity and widening trained models. Numerical results further corroborate our analytical findings.