Problems With Evaluation of Word Embeddings Using Word Similarity Tasks

TL;DR

This paper critically examines the limitations of using word similarity tasks for evaluating word embeddings, highlighting issues and advocating for the development of more reliable evaluation methods.

Contribution

It identifies key problems with current word similarity evaluations and summarizes potential solutions, emphasizing the need for improved evaluation standards.

Findings

Word similarity tasks are unreliable for evaluating word vectors.

Current evaluation methods lack standardization and robustness.

The paper calls for new approaches to intrinsic evaluation of word embeddings.

Abstract

Lacking standardized extrinsic evaluation methods for vector representations of words, the NLP community has relied heavily on word similarity tasks as a proxy for intrinsic evaluation of word vectors. Word similarity evaluation, which correlates the distance between vectors and human judgments of semantic similarity is attractive, because it is computationally inexpensive and fast. In this paper we present several problems associated with the evaluation of word vectors on word similarity datasets, and summarize existing solutions. Our study suggests that the use of word similarity tasks for evaluation of word vectors is not sustainable and calls for further research on evaluation methods.