Historical Test-time Prompt Tuning for Vision Foundation Models

TL;DR

This paper introduces HisTPT, a novel test-time prompt tuning method that uses multiple knowledge banks to memorize useful information, improving robustness and performance across various vision tasks and domain shifts.

Contribution

HisTPT is the first approach to incorporate multiple knowledge banks with adaptive retrieval for robust test-time prompt tuning in vision models.

Findings

Achieves superior performance across image classification, segmentation, and detection.

Effectively handles continuous domain shifts during testing.

Outperforms existing test-time prompt tuning methods.

Abstract

Test-time prompt tuning, which learns prompts online with unlabelled test samples during the inference stage, has demonstrated great potential by learning effective prompts on-the-fly without requiring any task-specific annotations. However, its performance often degrades clearly along the tuning process when the prompts are continuously updated with the test data flow, and the degradation becomes more severe when the domain of test samples changes continuously. We propose HisTPT, a Historical Test-time Prompt Tuning technique that memorizes the useful knowledge of the learnt test samples and enables robust test-time prompt tuning with the memorized knowledge. HisTPT introduces three types of knowledge banks, namely, local knowledge bank, hard-sample knowledge bank, and global knowledge bank, each of which works with different mechanisms for effective knowledge memorization and test-time prompt optimization. In addition, HisTPT features an adaptive knowledge retrieval mechanism that regularizes the prediction of each test sample by adaptively retrieving the memorized knowledge. Extensive experiments show that HisTPT achieves superior prompt tuning performance consistently while handling different visual recognition tasks (e.g., image classification, semantic segmentation, and object detection) and test samples from continuously changing domains.