Just Rank: Rethinking Evaluation with Word and Sentence Similarities

TL;DR

This paper critiques current semantic similarity evaluations for embeddings, introduces EvalRank as a more effective intrinsic evaluation method, and provides a comprehensive toolkit validated by extensive experiments.

Contribution

It proposes EvalRank, a novel intrinsic evaluation method that correlates better with downstream tasks, and releases a practical toolkit for future benchmarking.

Findings

EvalRank shows stronger correlation with downstream performance.

Existing similarity-based evaluations can mislead embedding development.

Extensive experiments validate the effectiveness of EvalRank across models and datasets.

Abstract

Word and sentence embeddings are useful feature representations in natural language processing. However, intrinsic evaluation for embeddings lags far behind, and there has been no significant update since the past decade. Word and sentence similarity tasks have become the de facto evaluation method. It leads models to overfit to such evaluations, negatively impacting embedding models' development. This paper first points out the problems using semantic similarity as the gold standard for word and sentence embedding evaluations. Further, we propose a new intrinsic evaluation method called EvalRank, which shows a much stronger correlation with downstream tasks. Extensive experiments are conducted based on 60+ models and popular datasets to certify our judgments. Finally, the practical evaluation toolkit is released for future benchmarking purposes.