Model Choices Influence Attributive Word Associations: A Semi-supervised Analysis of Static Word Embeddings

TL;DR

This study investigates how different static word embedding architectures and training choices influence the encoding of attributive word associations, revealing significant model-dependent variations and sensitivities.

Contribution

It introduces a semi-supervised clustering approach to analyze attributive word associations across multiple embedding models, highlighting the impact of training procedures and corpora.

Findings

Context learning flavor affects association distinguishability

Significant inter-model disparity in word associations

Embedding architecture influences association patterns

Abstract

Static word embeddings encode word associations, extensively utilized in downstream NLP tasks. Although prior studies have discussed the nature of such word associations in terms of biases and lexical regularities captured, the variation in word associations based on the embedding training procedure remains in obscurity. This work aims to address this gap by assessing attributive word associations across five different static word embedding architectures, analyzing the impact of the choice of the model architecture, context learning flavor and training corpora. Our approach utilizes a semi-supervised clustering method to cluster annotated proper nouns and adjectives, based on their word embedding features, revealing underlying attributive word associations formed in the embedding space, without introducing any confirmation bias. Our results reveal that the choice of the context learning flavor during embedding training (CBOW vs skip-gram) impacts the word association distinguishability and word embeddings' sensitivity to deviations in the training corpora. Moreover, it is empirically shown that even when trained over the same corpora, there is significant inter-model disparity and intra-model similarity in the encoded word associations across different word embedding models, portraying specific patterns in the way the embedding space is created for each embedding architecture.