Modeling cognitive processes of natural reading with transformer-based Language Models

TL;DR

This paper evaluates transformer-based language models like GPT2 and LLaMA in explaining eye movement patterns during reading, showing they outperform older models but still do not fully replicate human predictability effects.

Contribution

The study extends previous research by assessing transformer models' ability to predict gaze durations, revealing their improved performance yet persistent limitations in modeling human reading behavior.

Findings

Transformer models outperform earlier models in explaining gaze duration variance.

Models still do not fully account for human predictability effects.

State-of-the-art models differ from human language processing.

Abstract

Recent advances in Natural Language Processing (NLP) have led to the development of highly sophisticated language models for text generation. In parallel, neuroscience has increasingly employed these models to explore cognitive processes involved in language comprehension. Previous research has shown that models such as N-grams and LSTM networks can partially account for predictability effects in explaining eye movement behaviors, specifically Gaze Duration, during reading. In this study, we extend these findings by evaluating transformer-based models (GPT2, LLaMA-7B, and LLaMA2-7B) to further investigate this relationship. Our results indicate that these architectures outperform earlier models in explaining the variance in Gaze Durations recorded from Rioplantense Spanish readers. However, similar to previous studies, these models still fail to account for the entirety of the variance captured by human predictability. These findings suggest that, despite their advancements, state-of-the-art language models continue to predict language in ways that differ from human readers.