Data Heterogeneity and Forgotten Labels in Split Federated Learning

TL;DR

This paper investigates catastrophic forgetting in Split Federated Learning caused by data heterogeneity and processing order, proposing Hydra, a novel method that improves model performance by mitigating forgetting.

Contribution

The paper identifies the forgetting phenomenon in SFL due to data heterogeneity and introduces Hydra, a new mitigation technique inspired by multi-head networks.

Findings

Hydra outperforms existing baselines in experiments.

Data heterogeneity causes class-specific forgetting in SFL.

Processing order at the server influences model forgetting.

Abstract

In Split Federated Learning (SFL), the clients collaboratively train a model with the help of a server by splitting the model into two parts. Part-1 is trained locally at each client and aggregated by the aggregator at the end of each round. Part-2 is trained at a server that sequentially processes the intermediate activations received from each client. We study the phenomenon of catastrophic forgetting (CF) in SFL in the presence of data heterogeneity. In detail, due to the nature of SFL, local updates of part-1 may drift away from global optima, while part-2 is sensitive to the processing sequence, similar to forgetting in continual learning (CL). Specifically, we observe that the trained model performs better in classes (labels) seen at the end of the sequence. We investigate this phenomenon with emphasis on key aspects of SFL, such as the processing order at the server and the cut layer. Based on our findings, we propose Hydra, a novel mitigation method inspired by multi-head neural networks and adapted for the SFL setting. Extensive numerical evaluations show that Hydra outperforms baselines and methods from the literature.