Bio-inspired Structure Identification in Language Embeddings

TL;DR

This paper introduces a bio-inspired approach to explore and visualize the geometric structure of word embeddings, revealing meaningful patterns and alternative similarity measures that differ from traditional metrics.

Contribution

It presents a novel bio-inspired methodology for analyzing the structure of language embeddings and comparing different embedding techniques.

Findings

Bio-inspired visualization reveals discernible structure in embeddings.

Alternative similarity rankings differ from cosine and Euclidean metrics.

Method helps investigate semantic interpretation variations.

Abstract

Word embeddings are a popular way to improve downstream performances in contemporary language modeling. However, the underlying geometric structure of the embedding space is not well understood. We present a series of explorations using bio-inspired methodology to traverse and visualize word embeddings, demonstrating evidence of discernible structure. Moreover, our model also produces word similarity rankings that are plausible yet very different from common similarity metrics, mainly cosine similarity and Euclidean distance. We show that our bio-inspired model can be used to investigate how different word embedding techniques result in different semantic outputs, which can emphasize or obscure particular interpretations in textual data.