# Pitfalls in the Evaluation of Sentence Embeddings

## TL;DR

This paper identifies key pitfalls in evaluating sentence embeddings, such as size comparison issues and normalization effects, and offers recommendations for more reliable future assessments.

## Contribution

It systematically highlights evaluation pitfalls in sentence embeddings and proposes improved practices to enhance assessment reliability.

## Key findings

- Comparison of embeddings of different sizes can be misleading
- Normalization significantly impacts evaluation results
- Transfer and probing task correlations are often low and inconsistent

## Abstract

Deep learning models continuously break new records across different NLP tasks. At the same time, their success exposes weaknesses of model evaluation. Here, we compile several key pitfalls of evaluation of sentence embeddings, a currently very popular NLP paradigm. These pitfalls include the comparison of embeddings of different sizes, normalization of embeddings, and the low (and diverging) correlations between transfer and probing tasks. Our motivation is to challenge the current evaluation of sentence embeddings and to provide an easy-to-access reference for future research. Based on our insights, we also recommend better practices for better future evaluations of sentence embeddings.