Resource-Constrained Federated Continual Learning: What Does Matter?

TL;DR

This paper evaluates how resource constraints on edge devices impact federated continual learning, revealing that current methods perform poorly under realistic resource limitations and highlighting the need for more resource-efficient approaches.

Contribution

It provides a large-scale benchmark analyzing the performance of state-of-the-art FCL methods under resource constraints, exposing their limitations for real-world deployment.

Findings

Existing FCL methods fail under resource constraints.

Resource dependence hampers FCL performance.

Insights for future resource-efficient FCL research.

Abstract

Federated Continual Learning (FCL) aims to enable sequentially privacy-preserving model training on streams of incoming data that vary in edge devices by preserving previous knowledge while adapting to new data. Current FCL literature focuses on restricted data privacy and access to previously seen data while imposing no constraints on the training overhead. This is unreasonable for FCL applications in real-world scenarios, where edge devices are primarily constrained by resources such as storage, computational budget, and label rate. We revisit this problem with a large-scale benchmark and analyze the performance of state-of-the-art FCL approaches under different resource-constrained settings. Various typical FCL techniques and six datasets in two incremental learning scenarios (Class-IL and Domain-IL) are involved in our experiments. Through extensive experiments amounting to a total of over 1,000+ GPU hours, we find that, under limited resource-constrained settings, existing FCL approaches, with no exception, fail to achieve the expected performance. Our conclusions are consistent in the sensitivity analysis. This suggests that most existing FCL methods are particularly too resource-dependent for real-world deployment. Moreover, we study the performance of typical FCL techniques with resource constraints and shed light on future research directions in FCL.