EMTeC: A Corpus of Eye Movements on Machine-Generated Texts

TL;DR

EMTeC provides a comprehensive eye-movement corpus for studying how humans read machine-generated texts, including detailed data, model internals, and annotations, enabling diverse research on reading behavior and language models.

Contribution

This paper introduces EMTeC, a novel corpus combining eye-tracking data, model internals, and linguistic annotations for machine-generated texts, facilitating multifaceted research.

Findings

Provides detailed eye movement data on machine-generated texts

Includes model internals like attention and hidden states

Enables analysis of decoding strategies and text types

Abstract

The Eye Movements on Machine-Generated Texts Corpus (EMTeC) is a naturalistic eye-movements-while-reading corpus of 107 native English speakers reading machine-generated texts. The texts are generated by three large language models using five different decoding strategies, and they fall into six different text type categories. EMTeC entails the eye movement data at all stages of pre-processing, i.e., the raw coordinate data sampled at 2000 Hz, the fixation sequences, and the reading measures. It further provides both the original and a corrected version of the fixation sequences, accounting for vertical calibration drift. Moreover, the corpus includes the language models' internals that underlie the generation of the stimulus texts: the transition scores, the attention scores, and the hidden states. The stimuli are annotated for a range of linguistic features both at text and at word level. We anticipate EMTeC to be utilized for a variety of use cases such as, but not restricted to, the investigation of reading behavior on machine-generated text and the impact of different decoding strategies; reading behavior on different text types; the development of new pre-processing, data filtering, and drift correction algorithms; the cognitive interpretability and enhancement of language models; and the assessment of the predictive power of surprisal and entropy for human reading times. The data at all stages of pre-processing, the model internals, and the code to reproduce the stimulus generation, data pre-processing and analyses can be accessed via https://github.com/DiLi-Lab/EMTeC/.