Orthogonal Projection Subspace to Aggregate Online Prior-knowledge for Continual Test-time Adaptation

TL;DR

This paper introduces OoPk, a novel method for continual test-time adaptation that uses orthogonal projection to preserve source knowledge while adapting to new domains, improving performance in semantic segmentation tasks.

Contribution

The paper proposes a new pipeline, OoPk, combining orthogonal projection subspaces and online prior-knowledge aggregation to enhance continual test-time adaptation.

Findings

Outperforms previous CTTA methods in semantic segmentation benchmarks.

Effectively balances adaptation and knowledge preservation.

Reduces error accumulation during continual adaptation.

Abstract

Continual Test Time Adaptation (CTTA) is a task that requires a source pre-trained model to continually adapt to new scenarios with changing target distributions. Existing CTTA methods primarily focus on mitigating the challenges of catastrophic forgetting and error accumulation. Though there have been emerging methods based on forgetting adaptation with parameter-efficient fine-tuning, they still struggle to balance competitive performance and efficient model adaptation, particularly in complex tasks like semantic segmentation. In this paper, to tackle the above issues, we propose a novel pipeline, Orthogonal Projection Subspace to aggregate online Prior-knowledge, dubbed OoPk. Specifically, we first project a tuning subspace orthogonally which allows the model to adapt to new domains while preserving the knowledge integrity of the pre-trained source model to alleviate catastrophic forgetting. Then, we elaborate an online prior-knowledge aggregation strategy that employs an aggressive yet efficient image masking strategy to mimic potential target dynamism, enhancing the student model's domain adaptability. This further gradually ameliorates the teacher model's knowledge, ensuring high-quality pseudo labels and reducing error accumulation. We demonstrate our method with extensive experiments that surpass previous CTTA methods and achieve competitive performances across various continual TTA benchmarks in semantic segmentation tasks.