Caption-Driven Explorations: Aligning Image and Text Embeddings through Human-Inspired Foveated Vision

TL;DR

This paper introduces a new dataset and a zero-shot model for predicting human-like visual scanpaths during image captioning, improving understanding of attention in vision and AI.

Contribution

It presents CapMIT1003 dataset and NevaClip model, combining CLIP and NeVA algorithms for task-driven scanpath prediction, a novel approach in attention modeling.

Findings

NevaClip outperforms existing models in plausibility

CapMIT1003 enables studying attention during captioning

Zero-shot approach reduces need for training data

Abstract

Understanding human attention is crucial for vision science and AI. While many models exist for free-viewing, less is known about task-driven image exploration. To address this, we introduce CapMIT1003, a dataset with captions and click-contingent image explorations, to study human attention during the captioning task. We also present NevaClip, a zero-shot method for predicting visual scanpaths by combining CLIP models with NeVA algorithms. NevaClip generates fixations to align the representations of foveated visual stimuli and captions. The simulated scanpaths outperform existing human attention models in plausibility for captioning and free-viewing tasks. This research enhances the understanding of human attention and advances scanpath prediction models.