Human Sentence Processing: Recurrence or Attention?

TL;DR

This paper compares Transformer and RNN language models to human reading effort, finding Transformers better explain reading times and neural data, challenging traditional views of human sentence processing.

Contribution

It demonstrates that Transformer models outperform RNNs in modeling human reading effort, providing new insights into the mechanisms of sentence processing.

Findings

Transformers better explain self-paced reading times.

Transformers align more closely with neural activity during reading.

Results challenge the idea that human processing is solely recurrent and immediate.

Abstract

Recurrent neural networks (RNNs) have long been an architecture of interest for computational models of human sentence processing. The recently introduced Transformer architecture outperforms RNNs on many natural language processing tasks but little is known about its ability to model human language processing. We compare Transformer- and RNN-based language models' ability to account for measures of human reading effort. Our analysis shows Transformers to outperform RNNs in explaining self-paced reading times and neural activity during reading English sentences, challenging the widely held idea that human sentence processing involves recurrent and immediate processing and provides evidence for cue-based retrieval.