See Where You Read with Eye Gaze Tracking and Large Language Model

TL;DR

This paper introduces a system that combines eye gaze tracking and large language models to accurately track reading progress during linear and jump reading, improving user experience despite gaze measurement limitations.

Contribution

The paper presents a novel reading tracking system that integrates gaze error models and LLMs, enabling effective jump reading detection and paragraph highlighting.

Findings

Achieved 84% accuracy in jump reading tracking

Demonstrated reliable linear reading tracking in experiments

Improved reading efficiency and user experience in field tests

Abstract

Losing track of reading progress during line switching can be frustrating. Eye gaze tracking technology offers a potential solution by highlighting read paragraphs, aiding users in avoiding wrong line switches. However, the gap between gaze tracking accuracy (2-3 cm) and text line spacing (3-5 mm) makes direct application impractical. Existing methods leverage the linear reading pattern but fail during jump reading. This paper presents a reading tracking and highlighting system that supports both linear and jump reading. Based on experimental insights from the gaze nature study of 16 users, two gaze error models are designed to enable both jump reading detection and relocation. The system further leverages the large language model's contextual perception capability in aiding reading tracking. A reading tracking domain-specific line-gaze alignment opportunity is also exploited to enable dynamic and frequent calibration of the gaze results. Controlled experiments demonstrate reliable linear reading tracking, as well as 84% accuracy in tracking jump reading. Furthermore, real field tests with 18 volunteers demonstrated the system's effectiveness in tracking and highlighting read paragraphs, improving reading efficiency, and enhancing user experience.