Overcoming Poor Word Embeddings with Word Definitions

TL;DR

This paper proposes a method for natural language understanding models to use natural language definitions to improve reasoning about rare or unseen words, bridging the gap caused by poor pretrained embeddings.

Contribution

It introduces a model that leverages natural language definitions to enhance understanding of rare words, addressing limitations of pretrained embeddings.

Findings

Model recovers most performance gap caused by untrained words

Using definitions improves reasoning about rare words

Definitions help compensate for poor embeddings

Abstract

Modern natural language understanding models depend on pretrained subword embeddings, but applications may need to reason about words that were never or rarely seen during pretraining. We show that examples that depend critically on a rarer word are more challenging for natural language inference models. Then we explore how a model could learn to use definitions, provided in natural text, to overcome this handicap. Our model's understanding of a definition is usually weaker than a well-modeled word embedding, but it recovers most of the performance gap from using a completely untrained word.