Preventing Latent Rehearsal Decay in Online Continual SSL with SOLAR

TL;DR

This paper introduces SOLAR, a method to prevent latent space degradation in online continual self-supervised learning, improving stability and performance in non-stationary data streams.

Contribution

The paper proposes SOLAR, a novel approach that uses online proxies and an explicit overlap loss to manage stability-plasticity trade-off in OCSSL.

Findings

SOLAR achieves state-of-the-art results on OCSSL benchmarks.

It maintains high convergence speed and final performance.

The proposed metrics diagnose latent space degradation effectively.

Abstract

This paper explores Online Continual Self-Supervised Learning (OCSSL), a scenario in which models learn from continuous streams of unlabeled, non-stationary data, where methods typically employ replay and fast convergence is a central desideratum. We find that OCSSL requires particular attention to the stability-plasticity trade-off: stable methods (e.g. replay with Reservoir sampling) are able to converge faster compared to plastic ones (e.g. FIFO buffer), but incur in performance drops under certain conditions. We explain this collapse phenomenon with the Latent Rehearsal Decay hypothesis, which attributes it to latent space degradation under excessive stability of replay. We introduce two metrics (Overlap and Deviation) that diagnose latent degradation and correlate with accuracy declines. Building on these insights, we propose SOLAR, which leverages efficient online proxies of Deviation to guide buffer management and incorporates an explicit Overlap loss, allowing SOLAR to adaptively managing plasticity. Experiments demonstrate that SOLAR achieves state-of-the-art performance on OCSSL vision benchmarks, with both high convergence speed and final performance.