# Context encoders as a simple but powerful extension of word2vec

## TL;DR

This paper introduces context encoders (ConEc), an extension of word2vec that enhances word representations by incorporating local context, enabling better handling of polysemy and out-of-vocabulary words, demonstrated through improved NER performance.

## Contribution

The paper proposes context encoders (ConEc), a simple yet effective extension of word2vec that generates context-dependent embeddings for out-of-vocabulary words and words with multiple meanings.

## Key findings

- ConEc improves embeddings for OOV words.
- Enhanced embeddings lead to better NER performance.
- The approach is computationally efficient.

## Abstract

With a simple architecture and the ability to learn meaningful word embeddings efficiently from texts containing billions of words, word2vec remains one of the most popular neural language models used today. However, as only a single embedding is learned for every word in the vocabulary, the model fails to optimally represent words with multiple meanings. Additionally, it is not possible to create embeddings for new (out-of-vocabulary) words on the spot. Based on an intuitive interpretation of the continuous bag-of-words (CBOW) word2vec model's negative sampling training objective in terms of predicting context based similarities, we motivate an extension of the model we call context encoders (ConEc). By multiplying the matrix of trained word2vec embeddings with a word's average context vector, out-of-vocabulary (OOV) embeddings and representations for a word with multiple meanings can be created based on the word's local contexts. The benefits of this approach are illustrated by using these word embeddings as features in the CoNLL 2003 named entity recognition (NER) task.