Craft: Cross-modal Aligned Features Improve Robustness of Prompt Tuning

TL;DR

Craft introduces a cross-modal feature alignment method that enhances prompt tuning robustness in vision-language models by reducing overfitting and domain shift, leading to improved generalization across various tasks.

Contribution

The paper proposes a novel Cross-modal Aligned Feature Tuning (Craft) method that aligns text and image features to improve prompt tuning robustness and out-of-distribution performance.

Findings

Up to 6.1% improvement in Base-to-Novel generalization

Up to 5.8% improvement in group robustness

Up to 2.7% improvement in out-of-distribution tasks

Abstract

Prompt Tuning has emerged as a prominent research paradigm for adapting vision-language models to various downstream tasks. However, recent research indicates that prompt tuning methods often lead to overfitting due to limited training samples. In this paper, we propose a Cross-modal Aligned Feature Tuning (Craft) method to address this issue. Cross-modal alignment is conducted by first selecting anchors from the alternative domain and deriving relative representations of the embeddings for the selected anchors. Optimizing for a feature alignment loss over anchor-aligned text and image modalities creates a more unified text-image common space. Overfitting in prompt tuning also deteriorates model performance on out-of-distribution samples. To further improve the prompt model's robustness, we propose minimizing Maximum Mean Discrepancy (MMD) over the anchor-aligned feature spaces to mitigate domain shift. The experiment on four different prompt tuning structures consistently shows the improvement of our method, with increases of up to $6.1\%$ in the Base-to-Novel generalization task, $5.8\%$ in the group robustness task, and $2.7\%$ in the out-of-distribution tasks. The code will be available at https://github.com/Jingchensun/Craft