Personalized Image Descriptions from Attention Sequences

TL;DR

This paper introduces DEPER, a model that personalizes image descriptions by incorporating individual viewing behaviors and linguistic styles, significantly improving description quality and human alignment across diverse datasets.

Contribution

It presents a novel approach that explicitly models personalized viewing patterns in image description generation, enabling effective few-shot personalization without retraining.

Findings

Achieves 24% average improvement across datasets.

Models personalized attention for more human-aligned descriptions.

Enables few-shot personalization with a lightweight adapter.

Abstract

People can view the same image differently: they focus on different regions, objects, and details in varying orders and describe them in distinct linguistic styles. This leads to substantial variability in image descriptions. However, existing models for personalized image description focus on linguistic style alone, with no prior work leveraging individual viewing patterns. We address this gap by explicitly modeling personalized viewing behavior as a core factor in description generation. Our method, DEPER (DEscription-PERception persona encoder), learns a subject embedding that captures both linguistic style and viewing behavior, guided by an auxiliary attention-prediction task. A lightweight adapter aligns these embeddings with a frozen vision-language model, enabling few-shot personalization without retraining. Across four datasets spanning diverse viewing tasks and both short and detailed descriptions, DEPER achieves a 24% average improvement, showing that modeling personalized attention produces more human-aligned and high-quality descriptions. We posit that understanding how people see helps predict what they say; modeling human diversity in perception can improve both performance and human alignment in multimodal systems.