Skip Tuning: Pre-trained Vision-Language Models are Effective and Efficient Adapters Themselves

TL;DR

Skip Tuning is a new method for adapting pre-trained vision-language models that reduces computational complexity by skipping certain layers and class-wise features, outperforming prompt tuning and adapters.

Contribution

This work introduces Skip Tuning, a novel adaptation paradigm that improves transfer efficiency without extra modules or context vectors, unlike existing prompt tuning methods.

Findings

Skip Tuning outperforms prompt tuning and adapter-based methods.

It achieves better transferability and efficiency across various benchmarks.

The method reduces feature-gradient flow complexity effectively.

Abstract

Prompt tuning (PT) has long been recognized as an effective and efficient paradigm for transferring large pre-trained vision-language models (VLMs) to downstream tasks by learning a tiny set of context vectors. Nevertheless, in this work, we reveal that freezing the parameters of VLMs during learning the context vectors neither facilitates the transferability of pre-trained knowledge nor improves the memory and time efficiency significantly. Upon further investigation, we find that reducing both the length and width of the feature-gradient propagation flows of the full fine-tuning (FT) baseline is key to achieving effective and efficient knowledge transfer. Motivated by this, we propose Skip Tuning, a novel paradigm for adapting VLMs to downstream tasks. Unlike existing PT or adapter-based methods, Skip Tuning applies Layer-wise Skipping (LSkip) and Class-wise Skipping (CSkip) upon the FT baseline without introducing extra context vectors or adapter modules. Extensive experiments across a wide spectrum of benchmarks demonstrate the superior effectiveness and efficiency of our Skip Tuning over both PT and adapter-based methods. Code: https://github.com/Koorye/SkipTuning.