When Test-Time Adaptation Meets Self-Supervised Models

TL;DR

This paper explores test-time adaptation for self-supervised models, proposing a collaborative framework that enhances model performance without relying on source pretraining, validated across multiple benchmarks.

Contribution

It introduces a novel self-supervised test-time adaptation protocol and a collaborative learning framework combining SSL and TTA with contrastive learning and knowledge distillation.

Findings

Effective adaptation of SSL models without source pretraining

Improved performance on TTA benchmarks across diverse models

Competitive results validated on multiple self-supervised models

Abstract

Training on test-time data enables deep learning models to adapt to dynamic environmental changes, enhancing their practical applicability. Online adaptation from source to target domains is promising but it remains highly reliant on the performance of source pretrained model. In this paper, we investigate whether test-time adaptation (TTA) methods can continuously improve models trained via self-supervised learning (SSL) without relying on source pretraining. We introduce a self-supervised TTA protocol after observing that existing TTA approaches struggle when directly applied to self-supervised models with low accuracy on the source domain. Furthermore, we propose a collaborative learning framework that integrates SSL and TTA models, leveraging contrastive learning and knowledge distillation for stepwise representation refinement. We validate our method on diverse self-supervised models, including DINO, MoCo, and iBOT, across TTA benchmarks. Extensive experiments validate the effectiveness of our approach in SSL, showing that it achieves competitive performance even without source pretraining.