Discovering Universal Geometry in Embeddings with ICA

TL;DR

This paper uses ICA to reveal a universal semantic structure in embeddings, showing that intrinsic axes are consistent across languages, algorithms, and modalities, thereby deepening understanding of embedding representations.

Contribution

The study introduces a novel ICA-based method to uncover a universal semantic geometry in embeddings, highlighting consistent intrinsic axes across diverse data.

Findings

Semantic axes are consistent across languages and modalities.

Embeddings can be decomposed into interpretable intrinsic axes.

Universal geometric patterns are identified in embedding spaces.

Abstract

This study utilizes Independent Component Analysis (ICA) to unveil a consistent semantic structure within embeddings of words or images. Our approach extracts independent semantic components from the embeddings of a pre-trained model by leveraging anisotropic information that remains after the whitening process in Principal Component Analysis (PCA). We demonstrate that each embedding can be expressed as a composition of a few intrinsic interpretable axes and that these semantic axes remain consistent across different languages, algorithms, and modalities. The discovery of a universal semantic structure in the geometric patterns of embeddings enhances our understanding of the representations in embeddings.