CogniFNN: A Fuzzy Neural Network Framework for Cognitive Word Embedding Evaluation

TL;DR

CogniFNN introduces a fuzzy neural network framework that enhances the evaluation of word embeddings by leveraging cognitive data, providing more accurate assessments than existing methods.

Contribution

This paper presents the first use of fuzzy neural networks for evaluating English word embeddings against cognitive datasets, capturing non-linear and non-stationary features.

Findings

CogniFNN achieved lower prediction errors for GloVe and BERT embeddings.

It demonstrated higher significant ratios with random embeddings.

The framework offers more accurate and comprehensive evaluations.

Abstract

Word embeddings can reflect the semantic representations, and the embedding qualities can be comprehensively evaluated with human natural reading-related cognitive data sources. In this paper, we proposed the CogniFNN framework, which is the first attempt at using fuzzy neural networks to extract non-linear and non-stationary characteristics for evaluations of English word embeddings against the corresponding cognitive datasets. In our experiment, we used 15 human cognitive datasets across three modalities: EEG, fMRI, and eye-tracking, and selected the mean square error and multiple hypotheses testing as metrics to evaluate our proposed CogniFNN framework. Compared to the recent pioneer framework, our proposed CogniFNN showed smaller prediction errors of both context-independent (GloVe) and context-sensitive (BERT) word embeddings, and achieved higher significant ratios with randomly generated word embeddings. Our findings suggested that the CogniFNN framework could provide a more accurate and comprehensive evaluation of cognitive word embeddings. It will potentially be beneficial to the further word embeddings evaluation on extrinsic natural language processing tasks.