A visual observation on the geometry of UMAP projections of the difference vectors of antonym and synonym word pair embeddings

TL;DR

This paper investigates the geometric structure of word pair difference vectors in transformer embeddings, revealing a curious swirl pattern in UMAP projections that distinguishes antonyms from synonyms.

Contribution

It uncovers a novel geometric pattern in embedding difference vectors, providing insights into how antonymy and synonymy are represented in transformer models.

Findings

Discovery of a swirl pattern in UMAP projections of difference vectors

Distinction between antonym and synonym pairs based on geometric structure

Insights into the encoding of semantic opposites in transformer embeddings

Abstract

Antonyms, or opposites, are sometimes defined as \emph{word pairs that have all of the same contextually relevant properties but one}. Seeing how transformer models seem to encode concepts as directions, this begs the question if one can detect ``antonymity'' in the geometry of the embedding vectors of word pairs, especially based on their difference vectors. Such geometrical studies are then naturally contrasted by comparing antonymic pairs to their opposites; synonyms. This paper started as an exploratory project on the complexity of the systems needed to detect the geometry of the embedding vectors of antonymic word pairs. What we now report is a curious ``swirl'' that appears across embedding models in a somewhat specific projection configuration.