Uncovering divergent linguistic information in word embeddings with lessons for intrinsic and extrinsic evaluation

TL;DR

This paper demonstrates that word embeddings encode more information than apparent, and that linear transformations can enhance their linguistic properties, revealing insights into intrinsic and extrinsic evaluation methods.

Contribution

It introduces a linear transformation approach to adjust similarity orders in embeddings, improving their performance in capturing linguistic aspects without external resources.

Findings

Transformations improve intrinsic linguistic property capture.

Transformations have a greater impact on unsupervised downstream tasks.

Embeddings encode more diverse information than initially apparent.

Abstract

Following the recent success of word embeddings, it has been argued that there is no such thing as an ideal representation for words, as different models tend to capture divergent and often mutually incompatible aspects like semantics/syntax and similarity/relatedness. In this paper, we show that each embedding model captures more information than directly apparent. A linear transformation that adjusts the similarity order of the model without any external resource can tailor it to achieve better results in those aspects, providing a new perspective on how embeddings encode divergent linguistic information. In addition, we explore the relation between intrinsic and extrinsic evaluation, as the effect of our transformations in downstream tasks is higher for unsupervised systems than for supervised ones.