OledFL: Unleashing the Potential of Decentralized Federated Learning via Opposite Lookahead Enhancement

TL;DR

OledFL introduces an opposite lookahead technique to enhance decentralized federated learning, significantly improving convergence speed and generalization ability over existing methods, with theoretical guarantees and empirical validation.

Contribution

The paper proposes OledFL, a novel method that improves DFL by optimizing client initialization using opposite lookahead, with proven convergence and generalization bounds.

Findings

Achieves up to 5% performance improvement on CIFAR datasets.

Provides 8x faster convergence compared to DFedAvg.

Theoretically guarantees convergence rate and generalization bound.

Abstract

Decentralized Federated Learning (DFL) surpasses Centralized Federated Learning (CFL) in terms of faster training, privacy preservation, and light communication, making it a promising alternative in the field of federated learning. However, DFL still exhibits significant disparities with CFL in terms of generalization ability such as rarely theoretical understanding and degraded empirical performance due to severe inconsistency. In this paper, we enhance the consistency of DFL by developing an opposite lookahead enhancement technique (Ole), yielding OledFL to optimize the initialization of each client in each communication round, thus significantly improving both the generalization and convergence speed. Moreover, we rigorously establish its convergence rate in non-convex setting and characterize its generalization bound through uniform stability, which provides concrete reasons why OledFL can achieve both the fast convergence speed and high generalization ability. Extensive experiments conducted on the CIFAR10 and CIFAR100 datasets with Dirichlet and Pathological distributions illustrate that our OledFL can achieve up to 5\% performance improvement and 8$\times$ speedup, compared to the most popular DFedAvg optimizer in DFL.