Through Their Eyes: Fixation-aligned Tuning for Personalized User Emulation

TL;DR

This paper introduces FixATE, a method that aligns vision-language model attention with individual user gaze patterns to improve user simulation fidelity in recommendation systems.

Contribution

It proposes a novel fixation-aligned tuning approach that personalizes VLM attention to better emulate user behavior in visual recommendation interfaces.

Findings

Personalized gaze patterns are highly predictive of user clicks.

FixATE improves attention alignment with individual gaze data.

FixATE enhances click prediction accuracy across different models.

Abstract

Large language model (LLM) agents are increasingly deployed as scalable user simulators for recommender system evaluation. Yet existing simulators perceive recommendations through text or structured metadata rather than the visual interfaces real users browse-a critical gap, since attention over recommendation layouts is both visually driven and highly personalized. We investigate whether aligning a vision-language model's (VLM's) visual attention with user-specific gaze patterns can improve simulation fidelity. Analysis of a real-world eye-tracking dataset collected in a carousel-based recommendation setting reveals that users exhibit stable individual gaze patterns strongly predictive of click behavior. Building on this finding, we propose Fixation-Aligned Tuning for user Emulation (FixATE). Our approach first probes the VLM's internal visual attention via interpretability operators to obtain a slot-level relevance distribution comparable with human fixation, and then learns personalized soft prompts to steer the model's attention toward each user's characteristic fixation pattern. Experiments across three interpretability-based probing operators and two architecturally distinct VLM backbones demonstrate consistent improvements in both attention alignment and click prediction accuracy. These results suggest that making the model "see like the user" is a viable path toward simulators that more faithfully reproduce how users perceive and act in recommendation interfaces.