FedRewind: Rewinding Continual Model Exchange for Decentralized Federated Learning

TL;DR

FedRewind introduces a decentralized federated learning approach inspired by continual learning, where nodes rewind models to previous states to mitigate data distribution shifts, improving performance in dynamic, distributed environments.

Contribution

The paper proposes FedRewind, a novel decentralized federated learning method that incorporates model rewinding inspired by continual learning to handle spatial and temporal data shifts.

Findings

Outperforms standard decentralized federated learning methods.

Effectively reduces distribution shift through model rewinding.

Excels in federated continual learning scenarios with data shifts over space and time.

Abstract

In this paper, we present FedRewind, a novel approach to decentralized federated learning that leverages model exchange among nodes to address the issue of data distribution shift. Drawing inspiration from continual learning (CL) principles and cognitive neuroscience theories for memory retention, FedRewind implements a decentralized routing mechanism where nodes send/receive models to/from other nodes in the federation to address spatial distribution challenges inherent in distributed learning (FL). During local training, federation nodes periodically send their models back (i.e., rewind) to the nodes they received them from for a limited number of iterations. This strategy reduces the distribution shift between nodes' data, leading to enhanced learning and generalization performance. We evaluate our method on multiple benchmarks, demonstrating its superiority over standard decentralized federated learning methods and those enforcing specific routing schemes within the federation. Furthermore, the combination of federated and continual learning concepts enables our method to tackle the more challenging federated continual learning task, with data shifts over both space and time, surpassing existing baselines.