Topological Interpretations of GPT-3

TL;DR

This paper investigates the relationship between sentence vectors and semantic meaning across different embedding methods, revealing consistent correlation patterns that suggest underlying topological structures.

Contribution

It introduces a method to analyze the correlation between sentence embeddings and semantics using distance matrices across multiple embedding techniques.

Findings

Correlations of same sentence across different embeddings are consistent.

Correlations of different sentences within the same embedding space are observed.

The results support the hypothesis of underlying topological structures in sentence embeddings.

Abstract

This is an experiential study of investigating a consistent method for deriving the correlation between sentence vector and semantic meaning of a sentence. We first used three state-of-the-art word/sentence embedding methods including GPT-3, Word2Vec, and Sentence-BERT, to embed plain text sentence strings into high dimensional spaces. Then we compute the pairwise distance between any possible combination of two sentence vectors in an embedding space and map them into a matrix. Based on each distance matrix, we compute the correlation of distances of a sentence vector with respect to the other sentence vectors in an embedding space. Then we compute the correlation of each pair of the distance matrices. We observed correlations of the same sentence in different embedding spaces and correlations of different sentences in the same embedding space. These observations are consistent with our hypothesis and take us to the next stage.