Learning to Embed Words in Context for Syntactic Tasks

TL;DR

This paper introduces context-dependent token embedding models that improve syntactic task performance by capturing word sense and syntactic roles, trained on large unannotated corpora and tested on smaller annotated datasets.

Contribution

It presents simple neural network-based token embedding models that effectively encode context-specific word features for syntactic tasks, demonstrating improved performance over baselines.

Findings

Token embeddings outperform baseline models in syntactic tasks.

Models trained on large unannotated data improve small-data task performance.

Embedding models are efficient and adaptable across different context window sizes.

Abstract

We present models for embedding words in the context of surrounding words. Such models, which we refer to as token embeddings, represent the characteristics of a word that are specific to a given context, such as word sense, syntactic category, and semantic role. We explore simple, efficient token embedding models based on standard neural network architectures. We learn token embeddings on a large amount of unannotated text and evaluate them as features for part-of-speech taggers and dependency parsers trained on much smaller amounts of annotated data. We find that predictors endowed with token embeddings consistently outperform baseline predictors across a range of context window and training set sizes.