Big Data Small Data, In Domain Out-of Domain, Known Word Unknown Word: The Impact of Word Representation on Sequence Labelling Tasks

TL;DR

This paper evaluates five word embedding methods on sequence labelling tasks, showing embeddings improve performance with limited data, are robust out-of-domain, and are often comparable to simpler clustering methods.

Contribution

It provides an extensive extrinsic evaluation of multiple word embeddings across various NLP sequence labelling tasks, highlighting their effectiveness and robustness.

Findings

Few hundred training instances suffice for competitive results.

Word embeddings improve handling of OOV and out-of-domain words.

Simple Brown clusters often match the performance of complex embeddings.

Abstract

Word embeddings -- distributed word representations that can be learned from unlabelled data -- have been shown to have high utility in many natural language processing applications. In this paper, we perform an extrinsic evaluation of five popular word embedding methods in the context of four sequence labelling tasks: POS-tagging, syntactic chunking, NER and MWE identification. A particular focus of the paper is analysing the effects of task-based updating of word representations. We show that when using word embeddings as features, as few as several hundred training instances are sufficient to achieve competitive results, and that word embeddings lead to improvements over OOV words and out of domain. Perhaps more surprisingly, our results indicate there is little difference between the different word embedding methods, and that simple Brown clusters are often competitive with word embeddings across all tasks we consider.