From Local Details to Global Context: Advancing Vision-Language Models with Attention-Based Selection

TL;DR

This paper introduces an attention-guided cropping and feature selection method called ABS that enhances vision-language models' global understanding and zero-shot performance without additional training.

Contribution

The paper proposes a novel attention-based selection technique that improves global semantic understanding in vision-language models, achieving state-of-the-art results without training.

Findings

ABS outperforms previous methods on out-of-distribution tasks.

ABS rivals few-shot and test-time adaptation methods.

The approach is training-free and effective in zero-shot settings.

Abstract

Pretrained vision-language models (VLMs), e.g., CLIP, demonstrate impressive zero-shot capabilities on downstream tasks. Prior research highlights the crucial role of visual augmentation techniques, like random cropping, in alignment with fine-grained class descriptions generated by large language models (LLMs), significantly enhancing zero-shot performance by incorporating multi-view information. However, the inherent randomness of these augmentations can inevitably introduce background artifacts and cause models to overly focus on local details, compromising global semantic understanding. To address these issues, we propose an \textbf{A}ttention-\textbf{B}ased \textbf{S}election (\textbf{ABS}) method from local details to global context, which applies attention-guided cropping in both raw images and feature space, supplement global semantic information through strategic feature selection. Additionally, we introduce a soft matching technique to effectively filter LLM descriptions for better alignment. \textbf{ABS} achieves state-of-the-art performance on out-of-distribution generalization and zero-shot classification tasks. Notably, \textbf{ABS} is training-free and even rivals few-shot and test-time adaptation methods. Our code is available at \href{https://github.com/BIT-DA/ABS}{\textcolor{darkgreen}{https://github.com/BIT-DA/ABS}}.