Evaluation of sentence embeddings in downstream and linguistic probing tasks

TL;DR

This paper provides a comprehensive evaluation of recent sentence embedding methods across various downstream and linguistic tasks, revealing that simple bag-of-words approaches with deep context-dependent embeddings often outperform complex sentence encoders, yet universal performance remains elusive.

Contribution

It offers a thorough comparison of recent sentence embedding techniques and highlights the limitations of current models in achieving universal applicability.

Findings

Bag-of-words with deep context-dependent embeddings outperform some sentence encoders.

Current models are not yet universally effective across all tasks.

Simple approaches can be competitive with complex models.

Abstract

Despite the fast developmental pace of new sentence embedding methods, it is still challenging to find comprehensive evaluations of these different techniques. In the past years, we saw significant improvements in the field of sentence embeddings and especially towards the development of universal sentence encoders that could provide inductive transfer to a wide variety of downstream tasks. In this work, we perform a comprehensive evaluation of recent methods using a wide variety of downstream and linguistic feature probing tasks. We show that a simple approach using bag-of-words with a recently introduced language model for deep context-dependent word embeddings proved to yield better results in many tasks when compared to sentence encoders trained on entailment datasets. We also show, however, that we are still far away from a universal encoder that can perform consistently across several downstream tasks.