Imputing Out-of-Vocabulary Embeddings with LOVE Makes Language Models Robust with Little Cost

TL;DR

The paper introduces LOVE, a contrastive learning framework that enhances language models' robustness to out-of-vocabulary words by generating embeddings from surface forms, with minimal additional parameters.

Contribution

LOVE is a simple, lightweight method that extends pre-trained models to handle OOV words effectively, outperforming prior approaches in robustness and compatibility.

Findings

LOVE achieves comparable or better performance than existing methods.

It significantly improves robustness of BERT and FastText to OOV words.

The approach requires few additional parameters and is plug-and-play compatible.

Abstract

State-of-the-art NLP systems represent inputs with word embeddings, but these are brittle when faced with Out-of-Vocabulary (OOV) words. To address this issue, we follow the principle of mimick-like models to generate vectors for unseen words, by learning the behavior of pre-trained embeddings using only the surface form of words. We present a simple contrastive learning framework, LOVE, which extends the word representation of an existing pre-trained language model (such as BERT), and makes it robust to OOV with few additional parameters. Extensive evaluations demonstrate that our lightweight model achieves similar or even better performances than prior competitors, both on original datasets and on corrupted variants. Moreover, it can be used in a plug-and-play fashion with FastText and BERT, where it significantly improves their robustness.