Optimal synthesis embeddings

TL;DR

This paper presents a novel word embedding composition method that ensures fair and balanced representations for sets of words, applicable to static and contextualized embeddings, with theoretical foundations and practical evaluations.

Contribution

The paper introduces a new embedding composition technique based on distance minimization, with theoretical characterization and broad applicability to sentence and set representations.

Findings

Method outperforms baselines in probing linguistic features

Effective in data augmentation and sentence classification

Excels in capturing simple linguistic features

Abstract

In this paper we introduce a word embedding composition method based on the intuitive idea that a fair embedding representation for a given set of words should satisfy that the new vector will be at the same distance of the vector representation of each of its constituents, and this distance should be minimized. The embedding composition method can work with static and contextualized word representations, it can be applied to create representations of sentences and learn also representations of sets of words that are not necessarily organized as a sequence. We theoretically characterize the conditions for the existence of this type of representation and derive the solution. We evaluate the method in data augmentation and sentence classification tasks, investigating several design choices of embeddings and composition methods. We show that our approach excels in solving probing tasks designed to capture simple linguistic features of sentences.