Leveraging recent advances in Pre-Trained Language Models forEye-Tracking Prediction

TL;DR

This paper explores using pre-trained language models and eye-tracking data to predict gaze features during reading, proposing a novel hybrid neural architecture that improves prediction accuracy.

Contribution

It introduces a new neural network architecture combining RoBERTa and custom features for eye-tracking prediction, leveraging recent advances in language models.

Findings

The hybrid model outperforms individual models in MAE and R2 scores.

Feature engineering combined with pre-trained models enhances prediction accuracy.

The approach demonstrates the potential of language models in cognitive and linguistic tasks.

Abstract

Cognitively inspired Natural Language Pro-cessing uses human-derived behavioral datalike eye-tracking data, which reflect the seman-tic representations of language in the humanbrain to augment the neural nets to solve arange of tasks spanning syntax and semanticswith the aim of teaching machines about lan-guage processing mechanisms. In this paper,we use the ZuCo 1.0 and ZuCo 2.0 dataset con-taining the eye-gaze features to explore differ-ent linguistic models to directly predict thesegaze features for each word with respect to itssentence. We tried different neural networkmodels with the words as inputs to predict thetargets. And after lots of experimentation andfeature engineering finally devised a novel ar-chitecture consisting of RoBERTa Token Clas-sifier with a dense layer on top for languagemodeling and a stand-alone model consistingof dense layers followed by a transformer layerfor the extra features we engineered. Finally,we took the mean of the outputs of both thesemodels to make the final predictions. We eval-uated the models using mean absolute error(MAE) and the R2 score for each target.