What you can cram into a single vector: Probing sentence embeddings for linguistic properties

TL;DR

This paper introduces 10 probing tasks to analyze the linguistic properties captured by sentence embeddings, revealing insights into how different encoders and training methods encode linguistic information.

Contribution

It presents a new set of probing tasks for detailed analysis of sentence embeddings and compares multiple encoders and training strategies to understand their linguistic representations.

Findings

Different encoders capture distinct linguistic features

Training methods influence the type of information encoded

Probing tasks reveal nuanced properties of sentence embeddings

Abstract

Although much effort has recently been devoted to training high-quality sentence embeddings, we still have a poor understanding of what they are capturing. "Downstream" tasks, often based on sentence classification, are commonly used to evaluate the quality of sentence representations. The complexity of the tasks makes it however difficult to infer what kind of information is present in the representations. We introduce here 10 probing tasks designed to capture simple linguistic features of sentences, and we use them to study embeddings generated by three different encoders trained in eight distinct ways, uncovering intriguing properties of both encoders and training methods.