Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization

TL;DR

This paper analyzes the convergence issues caused by heterogeneity in federated optimization and introduces FedNova, a normalization method that addresses objective inconsistency and improves convergence behavior.

Contribution

It provides a unified framework for analyzing federated optimization algorithms and proposes FedNova, a novel normalization technique that eliminates objective inconsistency.

Findings

FedNova removes objective inconsistency in federated learning.

The analysis explains convergence slowdown due to heterogeneity.

FedNova maintains fast error convergence.

Abstract

In federated optimization, heterogeneity in the clients' local datasets and computation speeds results in large variations in the number of local updates performed by each client in each communication round. Naive weighted aggregation of such models causes objective inconsistency, that is, the global model converges to a stationary point of a mismatched objective function which can be arbitrarily different from the true objective. This paper provides a general framework to analyze the convergence of federated heterogeneous optimization algorithms. It subsumes previously proposed methods such as FedAvg and FedProx and provides the first principled understanding of the solution bias and the convergence slowdown due to objective inconsistency. Using insights from this analysis, we propose FedNova, a normalized averaging method that eliminates objective inconsistency while preserving fast error convergence.