Few-Shot Adversarial Low-Rank Fine-Tuning of Vision-Language Models
Sajjad Ghiasvand, Haniyeh Ehsani Oskouie, Mahnoosh Alizadeh, Ramtin Pedarsani
TL;DR
This paper introduces AdvCLIP-LoRA, a novel method for enhancing the adversarial robustness of vision-language models like CLIP during few-shot fine-tuning, achieving state-of-the-art results across multiple datasets.
Contribution
It is the first approach to combine low-rank adaptation with adversarial training for robust few-shot fine-tuning of CLIP models.
Findings
Achieves higher adversarial robustness than prompt tuning baselines.
Maintains competitive clean accuracy across datasets.
Demonstrates effectiveness on multiple backbones and datasets.
Abstract
Vision-Language Models (VLMs) such as CLIP have shown remarkable performance in cross-modal tasks through large-scale contrastive pre-training. To adapt these large transformer-based models efficiently for downstream tasks, Parameter-Efficient Fine-Tuning (PEFT) techniques like (Low-Rank Adaptation) LoRA have emerged as scalable alternatives to full fine-tuning, especially in few-shot scenarios. However, like traditional deep neural networks, VLMs are highly vulnerable to adversarial attacks, where imperceptible perturbations can significantly degrade model performance. Adversarial training remains the most effective strategy for improving model robustness in PEFT. In this work, we propose AdvCLIP-LoRA, to our knowledge the first method designed to enhance the adversarial robustness of CLIP models fine-tuned with LoRA in few-shot settings. Our method formulates training as a minimax…
Peer Reviews
Decision·ICLR 2026 Conference Withdrawn Submission
The paper has a clear problem framing about robust few-shot adaptation of CLIP with PEFT, filling a gap left by prompt-only approaches. The method is simple to implement and explained with theoretical analyses. The proposed method obtains strong gains in robustnss at low shots, with competitive clean accuracy. Theoretical analysis gives convergence to a stationary point.
Robustness is evaluated mainly against PGD in L-inifinty norm. There are also some other cases like L2-robustness and AutoAttack Robustness Evaluation. It seems that at higher shots, the robustness advantage under PGD narrows and sometimes trails the best prompt baseline on clean accuracy. Experiments focus on ViT-B/16 and ViT-B/32; larger or newer CLIP/SigLIP backbones are not explored. The convergence guarantee relies on smoothness/boundedness assumptions.
1. The application of combining LoRA with adversarial training for adversarial robustness in few-shot VLMs. 2. Excellent experimental performance, especially compared to prompt-tuning methods in the 1-shot setting.
1. Limited novelty. Related methods have already been studied in [1, 2, 3], e.g., [1] investigated a similar method, though not specifically in the few-shot setting. 2. Limited theoretical contribution. The convergence analysis relies on several strong assumptions, such as the guarantee that the low-rank matrices A and B remain bounded in each iteration (i.e., not exceeding constants cA and cB in line 286). However, the optimization process lacks explicit constraint on the matrix norms. Therefor
- The paper presents a systematic exploration of adversarial robustness in few-shot LoRA-based CLIP adaptation, offering a parameter-efficient alternative to adversarial prompt tuning that achieves superior results on both clean and robust accuracy. - The paper provides a nontrivial convergence guarantee for the minimax optimization, enhancing the methodological soundness and theoretical completeness of the work.
- The paper lacks depth. It mainly demonstrates the numerical advantages of LoRA-based adversarial fine-tuning without providing a clear explanation of why it performs better than prompt-tuning-based methods. As an attempt to apply adversarial fine-tuning to a new adaptation paradigm, the paper should comprehensively compare and discuss various fine-tuning frameworks (e.g., prompt tuning, adapter tuning, full fine-tuning) to strengthen its contribution. - The experiments are not sufficiently com
1. The paper is well-written and organized. The storyline is straightforward and intuitive. The methodology part is easy to follow. 2. The paper introduces theoretical analyses regarding the convergence of the proposed PEFT-based adversarial finetuning method. 3. According to experimental results (especially Tables 1&2), the proposed method shows a significant improvement beyond previous approaches.
1. The proposed method seems to be a simple combination of LoRA and standard adversarial fine-tuning (TeCoA) [a]. I find it hard to get some novel insights from the paper. Can authors explicitly show the difficulty of merging these two techniques? 2. In addition, the paper lacks experimental comparisons with previous adversarial fine-tuning approaches [b,c]. 3. In addition to LoRA, there also exist a lot of PEFT strategies; the author can also discuss and compare some of them, e.g., vision pro
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdversarial Robustness in Machine Learning · Model Reduction and Neural Networks · Multimodal Machine Learning Applications
MethodsContrastive Language-Image Pre-training