Human Attention during Goal-directed Reading Comprehension Relies on Task Optimization

TL;DR

This study shows that attention during goal-directed reading is driven by task optimization, with neural network models accurately predicting reading behavior and eye movements based on task-specific attention mechanisms.

Contribution

It demonstrates that attention allocation in reading can be explained by task-optimized neural network models, linking cognitive attention to computational principles.

Findings

Neural network attention weights predict reading times.

Eye-tracking reveals different attention patterns for text features and questions.

Task-specific neural networks accurately model reading behavior.

Abstract

The computational principles underlying attention allocation in complex goal-directed tasks remain elusive. Goal-directed reading, i.e., reading a passage to answer a question in mind, is a common real-world task that strongly engages attention. Here, we investigate what computational models can explain attention distribution in this complex task. We show that the reading time on each word is predicted by the attention weights in transformer-based deep neural networks (DNNs) optimized to perform the same reading task. Eye-tracking further reveals that readers separately attend to basic text features and question-relevant information during first-pass reading and rereading, respectively. Similarly, text features and question relevance separately modulate attention weights in shallow and deep DNN layers. Furthermore, when readers scan a passage without a question in mind, their reading time is predicted by DNNs optimized for a word prediction task. Therefore, attention during real-world reading can be interpreted as the consequence of task optimization.