Continual Self-Supervised Learning with Masked Autoencoders in Remote Sensing

TL;DR

This paper introduces CoSMAE, a novel continual self-supervised learning method using masked autoencoders for remote sensing, which effectively reduces catastrophic forgetting and improves generalization across sequential tasks.

Contribution

It proposes a new approach combining data mixup and model mixup knowledge distillation to enhance continual learning in remote sensing with masked autoencoders.

Findings

Achieves up to 4.94% improvement over state-of-the-art CL methods.

Effectively reduces catastrophic forgetting in remote sensing tasks.

Enhances generalization across sequential tasks.

Abstract

The development of continual learning (CL) methods, which aim to learn new tasks in a sequential manner from the training data acquired continuously, has gained great attention in remote sensing (RS). The existing CL methods in RS, while learning new tasks, enhance robustness towards catastrophic forgetting. This is achieved by using a large number of labeled training samples, which is costly and not always feasible to gather in RS. To address this problem, we propose a novel continual self-supervised learning method in the context of masked autoencoders (denoted as CoSMAE). The proposed CoSMAE consists of two components: i) data mixup; and ii) model mixup knowledge distillation. Data mixup is associated with retaining information on previous data distributions by interpolating images from the current task with those from the previous tasks. Model mixup knowledge distillation is associated with distilling knowledge from past models and the current model simultaneously by interpolating their model weights to form a teacher for the knowledge distillation. The two components complement each other to regularize the MAE at the data and model levels to facilitate better generalization across tasks and reduce the risk of catastrophic forgetting. Experimental results show that CoSMAE achieves significant improvements of up to 4.94% over state-of-the-art CL methods applied to MAE. Our code is publicly available at: https://git.tu-berlin.de/rsim/CoSMAE.