Attending Form and Context to Generate Specialized Out-of-VocabularyWords Representations

TL;DR

This paper introduces a novel neural network layer that generates context-aware, task-specific representations for out-of-vocabulary words, significantly improving multilingual tagging performance without pre-training.

Contribution

It presents a new contextual-compositional layer for OOV words that learns sentence-dependent representations without pre-training, enhancing multilingual NLP tagging models.

Findings

Achieved state-of-the-art results on Universal Dependencies Dataset 1.4.

Improved tagging performance across 23 languages.

Effectively handles OOV words without pre-trained embeddings.

Abstract

We propose a new contextual-compositional neural network layer that handles out-of-vocabulary (OOV) words in natural language processing (NLP) tagging tasks. This layer consists of a model that attends to both the character sequence and the context in which the OOV words appear. We show that our model learns to generate task-specific \textit{and} sentence-dependent OOV word representations without the need for pre-training on an embedding table, unlike previous attempts. We insert our layer in the state-of-the-art tagging model of \citet{plank2016multilingual} and thoroughly evaluate its contribution on 23 different languages on the task of jointly tagging part-of-speech and morphosyntactic attributes. Our OOV handling method successfully improves performances of this model on every language but one to achieve a new state-of-the-art on the Universal Dependencies Dataset 1.4.