Mapping Unseen Words to Task-Trained Embedding Spaces

TL;DR

This paper introduces a neural network approach to map initial word embeddings to task-specific spaces, improving handling of unseen words in dependency parsing and sentiment analysis.

Contribution

We propose a neural mapping method that enhances the use of initial embeddings for unseen words in supervised tasks, reducing errors and improving performance.

Findings

Improved dependency parsing accuracy on sentences with out-of-vocabulary words.

Enhanced sentiment analysis results using the mapping technique.

The method is general and applicable to various NLP tasks.

Abstract

We consider the supervised training setting in which we learn task-specific word embeddings. We assume that we start with initial embeddings learned from unlabelled data and update them to learn task-specific embeddings for words in the supervised training data. However, for new words in the test set, we must use either their initial embeddings or a single unknown embedding, which often leads to errors. We address this by learning a neural network to map from initial embeddings to the task-specific embedding space, via a multi-loss objective function. The technique is general, but here we demonstrate its use for improved dependency parsing (especially for sentences with out-of-vocabulary words), as well as for downstream improvements on sentiment analysis.