Visual Prompt-Agnostic Evolution

TL;DR

This paper introduces Prompt-Agnostic Evolution (PAE), a novel method to improve visual prompt tuning by modeling prompt dynamics, resulting in faster convergence and better accuracy across multiple vision tasks.

Contribution

We propose PAE, a frequency-domain and stability-inspired approach that enhances prompt evolution, accelerates training, and is compatible with various VPT methods without backbone changes.

Findings

PAE achieves 1.41x faster convergence on average.

PAE improves accuracy by 1-3% across 25 datasets.

PAE is prompt-agnostic, lightweight, and seamlessly integrates with existing VPT variants.

Abstract

Visual Prompt Tuning (VPT) adapts a frozen Vision Transformer (ViT) to downstream tasks by inserting a small number of learnable prompt tokens into the token sequence at each layer. However, we observe that existing VPT variants often suffer from unstable training dynamics, characterized by gradient oscillations. A layer-wise analysis reveals that shallow-layer prompts tend to stagnate early, while deeper-layer prompts exhibit high-variance oscillations, leading to cross-layer mismatch. These issues slow convergence and degrade final performance. To address these challenges, we propose Prompt-Agnostic Evolution ($\mathtt{PAE}$), which strengthens vision prompt tuning by explicitly modeling prompt dynamics. From a frequency-domain perspective, we initialize prompts in a task-aware direction by uncovering and propagating frequency shortcut patterns that the backbone inherently exploits for recognition. To ensure coherent evolution across layers, we employ a shared Koopman operator that imposes a global linear transformation instead of uncoordinated, layer-specific updates. Finally, inspired by Lyapunov stability theory, we introduce a regularizer that constrains error amplification during evolution. Extensive experiments show that $\mathtt{PAE}$ accelerates convergence with an average $1.41\times$ speedup and improves accuracy by 1-3% on 25 datasets across multiple downstream tasks. Beyond performance, $\mathtt{PAE}$ is prompt-agnostic and lightweight, and it integrates seamlessly with diverse VPT variants without backbone modification or inference-time changes.