SpecPL: Disentangling Spectral Granularity for Prompt Learning

TL;DR

SpecPL introduces a spectral perspective to prompt learning for vision-language models, disentangling visual signals into semantic and granular components with counterfactual supervision to improve fine-grained discrimination.

Contribution

It proposes a novel spectral approach using a frozen VAE and counterfactual granule training to enhance prompt learning in vision-language models.

Findings

Achieves state-of-the-art performance on 11 benchmarks.

Reaches a new harmonic-mean accuracy of 81.51%.

Effectively bridges the stability-generalization gap in prompt learning.

Abstract

Existing prompt learning for VLMs exhibits a modality asymmetry, predominantly optimizing text tokens while still relying on frozen visual encoder as holistic extractor and neglecting the spectral granularity essential for fine-grained discrimination. To bridge this, we introduce Disentangling Spectral Granularity for Prompt Learning (SpecPL), which approaches prompt learning from a novel spectral perspective via Counterfactual Granule Supervision. Specifically, we leverage a frozen VAE to decompose visual signals into semantic low-frequency bands and granular high-frequency details. A frozen Visual Semantic Bank anchors text representations to universal low-frequency invariants, mitigating overfitting. Crucially, fine-grained discrimination is driven by counterfactual granule training: by permuting high-frequency signals, we compel the model to explicitly distinguish visual granularity from semantic invariance. Uniquely, SpecPL serves as a universal plug-and-play booster, revitalizing text-oriented baselines like CoOp and MaPLe via visual-side guidance. Experiments on 11 benchmarks demonstrate competitive state-of-the-art performance, achieving a new performance ceiling of 81.51\% harmonic-mean accuracy. These results validate that spectral disentanglement with counterfactual supervision effectively bridges the gap in the stability-generalization trade-off. Code is released at https://github.com/Mlrac1e/SpecPL-Prompt-Learning.