Can Eye Movement Data Be Used As Ground Truth For Word Embeddings Evaluation?

TL;DR

This paper investigates whether eye movement data during silent reading can serve as a reliable ground truth for evaluating distributional semantic models across English and Russian, finding that the hypothesis is questionable.

Contribution

It introduces a method to compare eye movement data with word embeddings across languages, challenging the assumption that eye data can universally evaluate semantic models.

Findings

Limited correlation between eye movement data and word embeddings.

Language independence of eye movement data as an evaluation standard is questionable.

Results suggest the need for alternative evaluation methods.

Abstract

In recent years a certain success in the task of modeling lexical semantics was obtained with distributional semantic models. Nevertheless, the scientific community is still unaware what is the most reliable evaluation method for these models. Some researchers argue that the only possible gold standard could be obtained from neuro-cognitive resources that store information about human cognition. One of such resources is eye movement data on silent reading. The goal of this work is to test the hypothesis of whether such data could be used to evaluate distributional semantic models on different languages. We propose experiments with English and Russian eye movement datasets (Provo Corpus, GECO and Russian Sentence Corpus), word vectors (Skip-Gram models trained on national corpora and Web corpora) and word similarity datasets of Russian and English assessed by humans in order to find the existence of correlation between embeddings and eye movement data and test the hypothesis that this correlation is language independent. As a result, we found that the validity of the hypothesis being tested could be questioned.