Masked Autoencoders are Parameter-Efficient Federated Continual Learners

TL;DR

This paper introduces pMAE, a parameter-efficient federated continual learning method using masked autoencoders, which effectively mitigates catastrophic forgetting and non-IID data issues in federated learning scenarios.

Contribution

The paper proposes pMAE, a novel federated continual learning approach leveraging masked autoencoders to improve data privacy and learning efficiency across clients.

Findings

pMAE achieves performance comparable to existing prompt-based methods.

pMAE enhances effectiveness when using self-supervised pre-trained transformers.

Experimental results validate pMAE's ability to mitigate catastrophic forgetting and non-IID issues.

Abstract

Federated learning is a specific distributed learning paradigm in which a central server aggregates updates from multiple clients' local models, thereby enabling the server to learn without requiring clients to upload their private data, maintaining data privacy. While existing federated learning methods are primarily designed for static data, real-world applications often require clients to learn new categories over time. This challenge necessitates the integration of continual learning techniques, leading to federated continual learning (FCL). To address both catastrophic forgetting and non-IID issues, we propose to use masked autoencoders (MAEs) as parameter-efficient federated continual learners, called pMAE. pMAE learns reconstructive prompt on the client side through image reconstruction using MAE. On the server side, it reconstructs the uploaded restore information to capture the data distribution across previous tasks and different clients, using these reconstructed images to fine-tune discriminative prompt and classifier parameters tailored for classification, thereby alleviating catastrophic forgetting and non-IID issues on a global scale. Experimental results demonstrate that pMAE achieves performance comparable to existing prompt-based methods and can enhance their effectiveness, particularly when using self-supervised pre-trained transformers as the backbone. Code is available at: https://github.com/ycheoo/pMAE.