From Question to Exploration: Test-Time Adaptation in Semantic Segmentation?

TL;DR

This paper investigates the challenges of applying test-time adaptation techniques from classification to semantic segmentation, revealing limitations of existing methods and proposing a new promising approach called TTAP.

Contribution

The paper provides a comprehensive analysis of classic TTA strategies in segmentation, identifies key challenges, and introduces TTAP, a novel method that outperforms existing techniques.

Findings

Classic normalization strategies have limited or negative impact on segmentation TTA.

Teacher-student schemes stabilize training but do not improve performance under complex distributions.

Segmentation TTA faces severe long-tailed class imbalance issues.

Abstract

Test-time adaptation (TTA) aims to adapt a model, initially trained on training data, to test data with potential distribution shifts. Most existing TTA methods focus on classification problems. The pronounced success of classification might lead numerous newcomers and engineers to assume that classic TTA techniques can be directly applied to the more challenging task of semantic segmentation. However, this belief is still an open question. In this paper, we investigate the applicability of existing classic TTA strategies in semantic segmentation. Our comprehensive results have led to three key observations. First, the classic normalization updating strategy only brings slight performance improvement, and in some cases, it might even adversely affect the results. Even with the application of advanced distribution estimation techniques like batch renormalization, the problem remains unresolved. Second, although the teacher-student scheme does enhance the training stability for segmentation TTA in the presence of noisy pseudo-labels and temporal correlation, it cannot directly result in performance improvement compared to the original model without TTA under complex data distribution. Third, segmentation TTA suffers a severe long-tailed class-imbalance problem, which is substantially more complex than that in TTA for classification. This long-tailed challenge negatively affects segmentation TTA performance, even when the accuracy of pseudo-labels is high. Besides those observations, we find that visual prompt tuning (VisPT) is promising in segmentation TTA and propose a novel method named TTAP. The outstanding performance of TTAP has also been verified. We hope the community can give more attention to this challenging, yet important, segmentation TTA task in the future. The source code is available at: \textit{https://github.com/ycarobot/TTAP