DynaPrompt: Dynamic Test-Time Prompt Tuning
Zehao Xiao, Shilin Yan, Jack Hong, Jiayin Cai, Xiaolong Jiang, Yao Hu,, Jiayi Shen, Qi Wang, Cees G. M. Snoek
TL;DR
DynaPrompt is a dynamic test-time prompt tuning method that adaptively selects and updates prompts based on test sample relevance, improving zero-shot generalization of vision-language models while mitigating error accumulation.
Contribution
It introduces a novel dynamic prompt selection and appending strategy that leverages test data distribution information during inference.
Findings
Effective on fourteen datasets
Reduces prompt collapse and error accumulation
Enhances zero-shot generalization
Abstract
Test-time prompt tuning enhances zero-shot generalization of vision-language models but tends to ignore the relatedness among test samples during inference. Online test-time prompt tuning provides a simple way to leverage the information in previous test samples, albeit with the risk of prompt collapse due to error accumulation. To enhance test-time prompt tuning, we propose DynaPrompt, short for dynamic test-time prompt tuning, exploiting relevant data distribution information while reducing error accumulation. Built on an online prompt buffer, DynaPrompt adaptively selects and optimizes the relevant prompts for each test sample during tuning. Specifically, we introduce a dynamic prompt selection strategy based on two metrics: prediction entropy and probability difference. For unseen test data information, we develop dynamic prompt appending, which allows the buffer to append new…
Peer Reviews
Decision·ICLR 2025 Poster
1. The paper is well-organized and easy to follow. 2. The proposed DynaPrompt effectively mitigates error accumulation, a prevalent challenge in online test-time tuning，leading to more stable performance across sequential test samples while exploiting beneficial information from prior online test samples. 3. Despite the increased time costs associated with larger prompt buffer sizes, the experimental outcomes confirm the effectiveness of the proposed method.
1. In prompt learning, the initial prompts might affect the final performance. I wonder whether a similar situation can occur with the proposed method. The authors are encouraged to conduct related experiments. 2. Could the proposed method be extended to incorporate visual prompts, thereby evolving into a multimodal approach? Additionally, when integrated with MaPLe, is the method only applied to the textual branch?
1. The paper is well-structured and easy to follow, with the three technical components clearly and accessibly presented. 2. The proposed method is well-reasoned, employing dynamic prompt selection and updating mechanisms that are both effective and distinct from prior studies, which primarily focus on data manipulation. 3. The experiments are thorough, and the results convincingly demonstrate the effectiveness of the proposed method.
1. The range of comparison methods could be expanded, as the paper overlooks one relevant comparison method [1]. 2. Both the prediction entropy metric and probability difference metric provide insights into model prediction confidence, though from different perspectives. It is unclear why entropy is specifically used to measure relevance while difference is used to maintain diversity. Why does the direct combination of these two types of prompts yield effective results? Would a two-step prompt s
1. This paper propose a dynamic prompting method that utilizes useful information from online test samples while mitigating the problem of error accumulation. 2. DynaPrompt improves the adaptive capability of the model by introducing a dynamic prompt selection strategy that adaptively selects and optimizes relevant prompts for each test sample based on two metrics: predictive entropy and probability difference. 3. During the dynamic prompt selection process, if no suitable prompts can be found,
1. The computational complexity increases, and the authors' approach requires dynamic updating and selection of cues at each stage, whether it introduces more computational time. 2. The authors' approach demonstrates the advantages of dynamic prompting, however did the authors consider whether comparable performance could also be achieved if online testing was performed from scratch using a prompt length comparable to that of the final model.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsVLSI and Analog Circuit Testing · Engineering and Test Systems · Software Testing and Debugging Techniques