Mechanistic Decomposition of Sentence Representations

TL;DR

This paper introduces a method to decompose sentence embeddings into interpretable components using dictionary learning, revealing that many semantic and syntactic features are linearly encoded within the embeddings.

Contribution

It presents a novel approach to mechanistically interpret sentence embeddings by linking token-level features to sentence-level representations.

Findings

Many semantic and syntactic aspects are linearly encoded in sentence embeddings

The method reveals how pooling compresses token features into sentence representations

The approach enhances transparency and controllability of sentence embeddings

Abstract

Sentence embeddings are central to modern NLP and AI systems, yet little is known about their internal structure. While we can compare these embeddings using measures such as cosine similarity, the contributing features are not human-interpretable, and the content of an embedding seems untraceable, as it is masked by complex neural transformations and a final pooling operation that combines individual token embeddings. To alleviate this issue, we propose a new method to mechanistically decompose sentence embeddings into interpretable components, by using dictionary learning on token-level representations. We analyze how pooling compresses these features into sentence representations, and assess the latent features that reside in a sentence embedding. This bridges token-level mechanistic interpretability with sentence-level analysis, making for more transparent and controllable representations. In our studies, we obtain several interesting insights into the inner workings of sentence embedding spaces, for instance, that many semantic and syntactic aspects are linearly encoded in the embeddings.