FALIP: Visual Prompt as Foveal Attention Boosts CLIP Zero-Shot Performance

TL;DR

FALIP introduces a train-free, attention-modulating method that enhances CLIP's zero-shot performance across various tasks by inserting foveal attention masks, avoiding alteration of original images.

Contribution

The paper presents FALIP, a novel train-free approach that adjusts CLIP's attention mechanism with foveal masks, improving zero-shot task performance without modifying images.

Findings

FALIP outperforms existing methods on most metrics.

FALIP boosts performance in referring expressions, classification, and 3D recognition.

FALIP can augment current methods for better results.

Abstract

CLIP has achieved impressive zero-shot performance after pre-training on a large-scale dataset consisting of paired image-text data. Previous works have utilized CLIP by incorporating manually designed visual prompts like colored circles and blur masks into the images to guide the model's attention, showing enhanced zero-shot performance in downstream tasks. Although these methods have achieved promising results, they inevitably alter the original information of the images, which can lead to failure in specific tasks. We propose a train-free method Foveal-Attention CLIP (FALIP), which adjusts the CLIP's attention by inserting foveal attention masks into the multi-head self-attention module. We demonstrate FALIP effectively boosts CLIP zero-shot performance in tasks such as referring expressions comprehension, image classification, and 3D point cloud recognition. Experimental results further show that FALIP outperforms existing methods on most metrics and can augment current methods to enhance their performance.