FALIP: Visual Prompt as Foveal Attention Boosts CLIP Zero-Shot Performance
Jiedong Zhuang, Jiaqi Hu, Lianrui Mu, Rui Hu, Xiaoyu Liang, Jiangnan, Ye, Haoji Hu

TL;DR
FALIP introduces a train-free, attention-modulating method that enhances CLIP's zero-shot performance across various tasks by inserting foveal attention masks, avoiding alteration of original images.
Contribution
The paper presents FALIP, a novel train-free approach that adjusts CLIP's attention mechanism with foveal masks, improving zero-shot task performance without modifying images.
Findings
FALIP outperforms existing methods on most metrics.
FALIP boosts performance in referring expressions, classification, and 3D recognition.
FALIP can augment current methods for better results.
Abstract
CLIP has achieved impressive zero-shot performance after pre-training on a large-scale dataset consisting of paired image-text data. Previous works have utilized CLIP by incorporating manually designed visual prompts like colored circles and blur masks into the images to guide the model's attention, showing enhanced zero-shot performance in downstream tasks. Although these methods have achieved promising results, they inevitably alter the original information of the images, which can lead to failure in specific tasks. We propose a train-free method Foveal-Attention CLIP (FALIP), which adjusts the CLIP's attention by inserting foveal attention masks into the multi-head self-attention module. We demonstrate FALIP effectively boosts CLIP zero-shot performance in tasks such as referring expressions comprehension, image classification, and 3D point cloud recognition. Experimental results…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsOlfactory and Sensory Function Studies · Retinal Development and Disorders · CCD and CMOS Imaging Sensors
MethodsSoftmax · Attention Is All You Need · Contrastive Language-Image Pre-training
