GLAD: Generalizable Tuning for Vision-Language Models

TL;DR

GLAD introduces a simple, effective, and generalizable tuning framework for vision-language models that improves robustness and transferability across various tasks and datasets by combining LoRA with gradient regularization.

Contribution

The paper proposes GLAD, a novel tuning method combining LoRA with gradient regularization, enhancing generalization and robustness in vision-language models.

Findings

GLAD outperforms previous methods on 15 benchmark datasets.

It improves base-to-novel class generalization.

It enhances image domain and cross-dataset generalization.

Abstract

Pre-trained vision-language models, such as CLIP, show impressive zero-shot recognition ability and can be easily transferred to specific downstream tasks via prompt tuning, even with limited training data. However, existing prompt tuning methods face two main challenges: (1) In few-shot scenarios, data scarcity often leads to overfitting, making the model sensitive to changes in the input domain. (2) To mitigate overfitting, these methods typically rely on complex task-specific model architectures and sensitive hyperparameter tuning, severely restricting their general applicability. To address these issues, we propose a simpler and more general framework called GLAD (Generalizable LoRA tuning with RegulArized GraDient). We show that merely applying LoRA achieves performance in downstream tasks comparable to current state-of-the-art prompt-based methods. While LoRA is effective and easy to use, it remains susceptible to overfitting in few-shot learning scenarios. To mitigate this risk, we introduce a gradient-based regularization technique. This technique effectively steers the optimization trajectory, encouraging the model to find a more stable parameter region that is robust to variations in data distribution. Through extensive experiments conducted on 15 benchmark datasets, we demonstrate that GLAD outperforms previous tuning approaches in terms of base-to-novel class generalization, image domain generalization, and cross-dataset generalization. The code will be publicly available.