Learning Dynamic Contextualised Word Embeddings via Template-based Temporal Adaptation

TL;DR

This paper introduces a novel approach for learning dynamic contextualized word embeddings by adapting pretrained language models with time-sensitive templates, capturing semantic changes over time without supervision.

Contribution

It presents an unsupervised method for selecting pivot and anchor terms, and automatic template learning, to effectively adapt language models to temporal data.

Findings

Reduces perplexity on test sentences in the target snapshot

Outperforms current state-of-the-art in dynamic embedding tasks

Effective for capturing semantic shifts over time

Abstract

Dynamic contextualised word embeddings (DCWEs) represent the temporal semantic variations of words. We propose a method for learning DCWEs by time-adapting a pretrained Masked Language Model (MLM) using time-sensitive templates. Given two snapshots $C_1$ and $C_2$ of a corpus taken respectively at two distinct timestamps $T_1$ and $T_2$, we first propose an unsupervised method to select (a) \emph{pivot} terms related to both $C_1$ and $C_2$, and (b) \emph{anchor} terms that are associated with a specific pivot term in each individual snapshot. We then generate prompts by filling manually compiled templates using the extracted pivot and anchor terms. Moreover, we propose an automatic method to learn time-sensitive templates from $C_1$ and $C_2$, without requiring any human supervision. Next, we use the generated prompts to adapt a pretrained MLM to $T_2$ by fine-tuning using those prompts. Multiple experiments show that our proposed method reduces the perplexity of test sentences in $C_2$, outperforming the current state-of-the-art.