Instance-Aware Test-Time Segmentation for Continual Domain Shifts

TL;DR

This paper introduces an instance-aware test-time adaptation method for semantic segmentation that dynamically adjusts pseudo labels based on confidence distributions, improving robustness across continual domain shifts.

Contribution

It presents a novel, fine-grained adaptation approach that considers class and instance variability, outperforming existing methods in continual domain shift scenarios.

Findings

Consistently outperforms state-of-the-art methods across multiple CTTA scenarios.

Effectively mitigates error accumulation during continual adaptation.

Enhances reliability of pseudo labels in semantic segmentation.

Abstract

Continual Test-Time Adaptation (CTTA) enables pre-trained models to adapt to continuously evolving domains. Existing methods have improved robustness but typically rely on fixed or batch-level thresholds, which cannot account for varying difficulty across classes and instances. This limitation is especially problematic in semantic segmentation, where each image requires dense, multi-class predictions. We propose an approach that adaptively adjusts pseudo labels to reflect the confidence distribution within each image and dynamically balances learning toward classes most affected by domain shifts. This fine-grained, class- and instance-aware adaptation produces more reliable supervision and mitigates error accumulation throughout continual adaptation. Extensive experiments across eight CTTA and TTA scenarios, including synthetic-to-real and long-term shifts, show that our method consistently outperforms state-of-the-art techniques, setting a new standard for semantic segmentation under evolving conditions.