Contextual Word Representations: A Contextual Introduction

TL;DR

This paper provides a broad overview of the development and significance of word representations in NLP, culminating in the discussion of recent advances in contextual word vectors, aimed at a general audience.

Contribution

It offers a non-technical, historical introduction to word embeddings and contextual word vectors, highlighting their evolution and open questions in NLP.

Findings

Explains the evolution of word vectors in NLP

Introduces the concept of contextual word vectors

Summarizes open research questions in the field

Abstract

This introduction aims to tell the story of how we put words into computers. It is part of the story of the field of natural language processing (NLP), a branch of artificial intelligence. It targets a wide audience with a basic understanding of computer programming, but avoids a detailed mathematical treatment, and it does not present any algorithms. It also does not focus on any particular application of NLP such as translation, question answering, or information extraction. The ideas presented here were developed by many researchers over many decades, so the citations are not exhaustive but rather direct the reader to a handful of papers that are, in the author's view, seminal. After reading this document, you should have a general understanding of word vectors (also known as word embeddings): why they exist, what problems they solve, where they come from, how they have changed over time, and what some of the open questions about them are. Readers already familiar with word vectors are advised to skip to Section 5 for the discussion of the most recent advance, contextual word vectors.