Continual Segmentation under Joint Nonstationarity

TL;DR

This paper addresses the challenge of continual semantic segmentation in evolving data streams with joint nonstationarity, proposing novel stabilization and semi-supervised techniques to improve robustness and performance.

Contribution

It introduces gradient-adaptive stabilization and prototype anchored supervision to handle coupled class, domain, and label shifts in continual segmentation.

Findings

Consistent improvements over prior methods in various regimes.

Reveals fundamental failure modes of existing approaches.

Provides insights into learning robust dense predictors in dynamic environments.

Abstract

Evolving data streams induce joint nonstationarity in continual semantic segmentation, where semantic classes, input distributions, and supervision availability change simultaneously over time. This setting reflects practical structured prediction systems, yet remains largely unexplored in prior continual learning work, which typically studies these factors in isolation. We formalize continual segmentation under coupled class, domain, and label shifts and investigate learning in heterogeneous dense prediction environments with limited annotations and abundant unlabeled data. To address instability and overfitting arising from few-shot supervision under distribution drift, we introduce gradient-adaptive stabilization, a parameter-wise regularization mechanism implemented via gradient-scaled stochastic perturbations that promotes a principled stability-plasticity tradeoff. We further leverage unlabeled data through semi-supervised learning and introduce prototype anchored supervision that validates pseudo-labels via joint confidence and prototype consistency. Together, these mechanisms enable learning under joint nonstationarity in continual segmentation. Extensive empirical evaluation across class-incremental, domain-incremental, and few-shot regimes demonstrates consistent improvements over prior methods in heterogeneous structured prediction settings. Our results expose fundamental failure modes of existing continual segmentation approaches and provide insight into learning robust dense predictors in dynamically evolving environments.