CogniVal: A Framework for Cognitive Word Embedding Evaluation

TL;DR

CogniVal introduces a multi-modal framework for evaluating English word embeddings based on cognitive signals like eye-tracking, EEG, and fMRI, providing a comprehensive assessment aligned with human semantic processing.

Contribution

This work is the first to propose a multi-modal, cognitively grounded evaluation framework for word embeddings, integrating diverse neural and behavioral data sources.

Findings

Strong correlations between cognitive datasets across modalities

Cognitive evaluation scores relate to NLP task performance

Framework is extensible to other evaluation methods

Abstract

An interesting method of evaluating word representations is by how much they reflect the semantic representations in the human brain. However, most, if not all, previous works only focus on small datasets and a single modality. In this paper, we present the first multi-modal framework for evaluating English word representations based on cognitive lexical semantics. Six types of word embeddings are evaluated by fitting them to 15 datasets of eye-tracking, EEG and fMRI signals recorded during language processing. To achieve a global score over all evaluation hypotheses, we apply statistical significance testing accounting for the multiple comparisons problem. This framework is easily extensible and available to include other intrinsic and extrinsic evaluation methods. We find strong correlations in the results between cognitive datasets, across recording modalities and to their performance on extrinsic NLP tasks.