Investigating the Contextualised Word Embedding Dimensions Specified for Contextual and Temporal Semantic Changes

TL;DR

This study analyzes how semantic changes are encoded in contextualised word embeddings, revealing that fine-tuning distributes change information across all dimensions and PCA better captures semantic shifts than ICA.

Contribution

It compares pre-trained and fine-tuned CWEs using PCA and ICA, showing PCA's superiority in representing semantic change axes and highlighting the distribution of change information.

Findings

Fewer axes are specific to semantic changes in pre-trained CWEs.

Fine-tuning distributes semantic change information across all dimensions.

PCA better captures semantic changes than ICA within top axes.

Abstract

The sense-aware contextualised word embeddings (SCWEs) encode semantic changes of words within the contextualised word embedding (CWE) spaces. Despite the superior performance of SCWEs in contextual/temporal semantic change detection (SCD) benchmarks, it remains unclear as to how the meaning changes are encoded in the embedding space. To study this, we compare pre-trained CWEs and their fine-tuned versions on contextual and temporal semantic change benchmarks under Principal Component Analysis (PCA) and Independent Component Analysis (ICA) transformations. Our experimental results reveal (a) although there exist a smaller number of axes that are specific to semantic changes of words in the pre-trained CWE space, this information gets distributed across all dimensions when fine-tuned, and (b) in contrast to prior work studying the geometry of CWEs, we find that PCA to better represent semantic changes than ICA within the top 10% of axes. These findings encourage the development of more efficient SCD methods with a small number of SCD-aware dimensions. Source code is available at https://github.com/LivNLP/svp-dims .