Dual-Stage Invariant Continual Learning under Extreme Visual Sparsity

TL;DR

This paper introduces a dual-stage invariant continual learning framework for object detection in extreme visual sparsity conditions, such as space-based RSO detection, addressing stability and adaptability challenges.

Contribution

It proposes a joint distillation approach enforcing structural and semantic consistency, along with a sparsity-aware data conditioning strategy for improved continual detection.

Findings

Achieved +4.0 mAP improvement over existing methods.

Demonstrated stability of backbone representations under sequential domain shifts.

Validated effectiveness on a high-resolution space-based RSO detection dataset.

Abstract

Continual learning seeks to maintain stable adaptation under non-stationary environments, yet this problem becomes particularly challenging in object detection, where most existing methods implicitly assume relatively balanced visual conditions. In extreme-sparsity regimes, such as those observed in space-based resident space object (RSO) detection scenarios, foreground signals are overwhelmingly dominated by background observations. Under such conditions, we analytically demonstrate that background-driven gradients destabilize the feature backbone during sequential domain shifts, causing progressive representation drift. This exposes a structural limitation of continual learning approaches relying solely on output-level distillation, as they fail to preserve intermediate representation stability. To address this, we propose a dual-stage invariant continual learning framework via joint distillation, enforcing structural and semantic consistency on both backbone representations and detection predictions, respectively, thereby suppressing error propagation at its source while maintaining adaptability. Furthermore, to regulate gradient statistics under severe imbalance, we introduce a sparsity-aware data conditioning strategy combining patch-based sampling and distribution-aware augmentation. Experiments on a high-resolution space-based RSO detection dataset show consistent improvement over established continual object detection methods, achieving an absolute gain of +4.0 mAP under sequential domain shifts.