GazeSummary: Exploring Gaze as an Implicit Prompt for Personalization in Text-based LLM Tasks

TL;DR

This paper explores using gaze tracking as an implicit prompt to personalize text-based tasks in LLMs, demonstrating that gaze data can improve summary quality and task support in realistic scenarios.

Contribution

It introduces a novel approach of leveraging gaze as an implicit prompt for personalization in LLMs, validated through experiments on realistic reading tasks.

Findings

Gaze data can be effectively used to personalize LLM outputs.

LLMs can generate high-quality summaries using gaze-based prompts.

Gaze-driven personalization enhances downstream task support.

Abstract

Smart glasses are accelerating progress toward more seamless and personalized LLM-based assistance by integrating multimodal inputs. Yet, these inputs rely on obtrusive explicit prompts. The advent of gaze tracking on smart devices offers a unique opportunity to extract implicit user intent for personalization. This paper investigates whether LLMs can interpret user gaze for text-based tasks. We evaluate different gaze representations for personalization and validate their effectiveness in realistic reading tasks. Results show that LLMs can leverage gaze to generate high-quality personalized summaries and support users in downstream tasks, highlighting the feasibility and value of gaze-driven personalization for future mobile and wearable LLM applications.