Benchmarking Catastrophic Forgetting Mitigation Methods in Federated Time Series Forecasting

TL;DR

This paper introduces a benchmarking framework for evaluating catastrophic forgetting mitigation methods in federated time series forecasting, addressing a gap in continual learning research for regression tasks in IoT applications.

Contribution

It presents the first benchmark for CF in federated time series forecasting, compares multiple mitigation strategies, and releases an open-source framework for future research.

Findings

Replay outperforms other methods in reducing forgetting.

Elastic Weight Consolidation shows moderate effectiveness.

The benchmark reveals significant variability in method performance.

Abstract

Catastrophic forgetting (CF) poses a persistent challenge in continual learning (CL), especially within federated learning (FL) environments characterized by non-i.i.d. time series data. While existing research has largely focused on classification tasks in vision domains, the regression-based forecasting setting prevalent in IoT and edge applications remains underexplored. In this paper, we present the first benchmarking framework tailored to investigate CF in federated continual time series forecasting. Using the Beijing Multi-site Air Quality dataset across 12 decentralized clients, we systematically evaluate several CF mitigation strategies, including Replay, Elastic Weight Consolidation, Learning without Forgetting, and Synaptic Intelligence. Key contributions include: (i) introducing a new benchmark for CF in time series FL, (ii) conducting a comprehensive comparative analysis of state-of-the-art methods, and (iii) releasing a reproducible open-source framework. This work provides essential tools and insights for advancing continual learning in federated time-series forecasting systems.