Fine-Grained Prediction of Reading Comprehension from Eye Movements

TL;DR

This paper investigates whether eye movement data can be used to accurately predict reading comprehension at a detailed level, using large-scale data and new multimodal models across different reading contexts.

Contribution

It introduces three novel multimodal language models for predicting comprehension from eye movements and evaluates their generalization across texts, participants, and reading regimes.

Findings

Eye movements contain useful signals for predicting comprehension.

Models show potential but face challenges in generalization.

The study provides a foundation for future research in eye-tracking-based comprehension assessment.

Abstract

Can human reading comprehension be assessed from eye movements in reading? In this work, we address this longstanding question using large-scale eyetracking data over textual materials that are geared towards behavioral analyses of reading comprehension. We focus on a fine-grained and largely unaddressed task of predicting reading comprehension from eye movements at the level of a single question over a passage. We tackle this task using three new multimodal language models, as well as a battery of prior models from the literature. We evaluate the models' ability to generalize to new textual items, new participants, and the combination of both, in two different reading regimes, ordinary reading and information seeking. The evaluations suggest that although the task is highly challenging, eye movements contain useful signals for fine-grained prediction of reading comprehension. Code and data will be made publicly available.