A La Carte Embedding: Cheap but Effective Induction of Semantic Feature Vectors

TL;DR

This paper presents a simple, efficient method called a la carte embedding for inducing semantic feature vectors for rare or unseen words and features, leveraging pretrained embeddings and linear regression.

Contribution

It introduces a novel linear transformation approach that enables quick, on-the-fly embedding induction for new textual features using minimal data, outperforming existing methods.

Findings

Requires fewer examples to learn high-quality embeddings

Achieves state-of-the-art results on nonce tasks

Effective for unsupervised document classification

Abstract

Motivations like domain adaptation, transfer learning, and feature learning have fueled interest in inducing embeddings for rare or unseen words, n-grams, synsets, and other textual features. This paper introduces a la carte embedding, a simple and general alternative to the usual word2vec-based approaches for building such representations that is based upon recent theoretical results for GloVe-like embeddings. Our method relies mainly on a linear transformation that is efficiently learnable using pretrained word vectors and linear regression. This transform is applicable on the fly in the future when a new text feature or rare word is encountered, even if only a single usage example is available. We introduce a new dataset showing how the a la carte method requires fewer examples of words in context to learn high-quality embeddings and we obtain state-of-the-art results on a nonce task and some unsupervised document classification tasks.