Dynamic Contextualized Word Embeddings

TL;DR

This paper introduces dynamic contextualized word embeddings that adapt to linguistic and extralinguistic contexts, leveraging pretrained language models to better capture semantic variability across time and social space.

Contribution

It proposes a novel embedding method that models words as functions of both linguistic and extralinguistic context, extending prior static and contextualized embeddings.

Findings

Effective in capturing semantic variability

Applicable to diverse NLP tasks

Demonstrated on four English datasets

Abstract

Static word embeddings that represent words by a single vector cannot capture the variability of word meaning in different linguistic and extralinguistic contexts. Building on prior work on contextualized and dynamic word embeddings, we introduce dynamic contextualized word embeddings that represent words as a function of both linguistic and extralinguistic context. Based on a pretrained language model (PLM), dynamic contextualized word embeddings model time and social space jointly, which makes them attractive for a range of NLP tasks involving semantic variability. We highlight potential application scenarios by means of qualitative and quantitative analyses on four English datasets.