MINT: Memory-Infused Prompt Tuning at Test-time for CLIP

TL;DR

MINT introduces a memory-augmented prompt tuning framework for vision-language models, enabling dynamic, test-time adaptation to distribution shifts by leveraging a learnable memory bank of prompt pairs, without retraining or source data.

Contribution

The paper proposes Memory-Infused Prompt Tuning (MINT), a novel method that dynamically retrieves and assembles prompts from a memory bank for improved test-time adaptation of VLMs.

Findings

Enhanced generalization under distribution shifts

Effective retrieval of relevant prompts during testing

No need for source data or retraining

Abstract

Improving the generalization ability of Vision-Language Pre-trained Models (VLMs) under test-time data distribution shifts remains a critical challenge. The existing Test-Time Adaptation (TTA) methods fall short in fully leveraging the model's internal knowledge, particularly in dynamically adapting to complex and hierarchical visual semantic information. In this paper, we propose Memory-Infused Prompt Tuning (MINT), a novel framework to address this issue. Inspired by human associative memory theory, MINT introduces a Memory Prompt Bank (MPB), which stores learnable key-value prompt pairs that work as a memory of previously seen samples. During the test time, relevant prompt pairs in the MPB are retrieved by the hierarchical visual features of test images to dynamically assemble Associative Prompts. The associative prompts are then injected into the image encoder for fine-grained, customized visual contextual guidance. MINT also utilizes learnable text prompts. MINT thus enables rapid, precise VLM adaptation at test time by leveraging this MPB-acquired memory, without source data or retraining. The code is available at https://github.com/Jamieyi2004/MINT.