Rehearsal-Free Continual Federated Learning with Synergistic Synaptic Intelligence

TL;DR

This paper introduces FedSSI, a regularization-based method for continual federated learning that avoids rehearsal, reduces computational costs, and effectively handles heterogeneous data distributions.

Contribution

The paper proposes FedSSI, a novel regularization algorithm tailored for heterogeneous data in continual federated learning, improving performance without rehearsal.

Findings

FedSSI outperforms existing methods in heterogeneous data scenarios.

Regularization techniques like synaptic intelligence can be adapted for CFL.

FedSSI reduces computational overhead compared to rehearsal-based approaches.

Abstract

Continual Federated Learning (CFL) allows distributed devices to collaboratively learn novel concepts from continuously shifting training data while avoiding knowledge forgetting of previously seen tasks. To tackle this challenge, most current CFL approaches rely on extensive rehearsal of previous data. Despite effectiveness, rehearsal comes at a cost to memory, and it may also violate data privacy. Considering these, we seek to apply regularization techniques to CFL by considering their cost-efficient properties that do not require sample caching or rehearsal. Specifically, we first apply traditional regularization techniques to CFL and observe that existing regularization techniques, especially synaptic intelligence, can achieve promising results under homogeneous data distribution but fail when the data is heterogeneous. Based on this observation, we propose a simple yet effective regularization algorithm for CFL named FedSSI, which tailors the synaptic intelligence for the CFL with heterogeneous data settings. FedSSI can not only reduce computational overhead without rehearsal but also address the data heterogeneity issue. Extensive experiments show that FedSSI achieves superior performance compared to state-of-the-art methods.