CapS-Adapter: Caption-based MultiModal Adapter in Zero-Shot Classification

TL;DR

CapS-Adapter introduces a caption-based multimodal support set method that leverages image and caption features to significantly improve zero-shot classification accuracy across diverse datasets without additional training.

Contribution

This work presents CapS-Adapter, a novel zero-shot classification approach using caption-based support sets to enhance generalization and performance over existing training-free methods.

Findings

Achieves 2.19% higher accuracy than previous state-of-the-art methods.

Demonstrates robust generalization across 19 benchmark datasets.

Effectively utilizes multimodal large models for support set construction.

Abstract

Recent advances in vision-language foundational models, such as CLIP, have demonstrated significant strides in zero-shot classification. However, the extensive parameterization of models like CLIP necessitates a resource-intensive fine-tuning process. In response, TIP-Adapter and SuS-X have introduced training-free methods aimed at bolstering the efficacy of downstream tasks. While these approaches incorporate support sets to maintain data distribution consistency between knowledge cache and test sets, they often fall short in terms of generalization on the test set, particularly when faced with test data exhibiting substantial distributional variations. In this work, we present CapS-Adapter, an innovative method that employs a caption-based support set, effectively harnessing both image and caption features to exceed existing state-of-the-art techniques in training-free scenarios. CapS-Adapter adeptly constructs support sets that closely mirror target distributions, utilizing instance-level distribution features extracted from multimodal large models. By leveraging CLIP's single and cross-modal strengths, CapS-Adapter enhances predictive accuracy through the use of multimodal support sets. Our method achieves outstanding zero-shot classification results across 19 benchmark datasets, improving accuracy by 2.19\% over the previous leading method. Our contributions are substantiated through extensive validation on multiple benchmark datasets, demonstrating superior performance and robust generalization capabilities. Our code is made publicly available at https://github.com/WLuLi/CapS-Adapter.