Decoding Dense Embeddings: Sparse Autoencoders for Interpreting and Discretizing Dense Retrieval

TL;DR

This paper introduces a framework using Sparse Autoencoders to interpret dense embeddings from DPR models by decomposing them into meaningful concepts, and proposes CL-SR for efficient, transparent retrieval.

Contribution

The work presents a novel interpretability method for dense embeddings and a new retrieval framework that combines semantic richness with efficiency and transparency.

Findings

CL-SR achieves high efficiency in index space and computation.

The framework provides human-interpretable concepts for dense embeddings.

CL-SR maintains robust retrieval performance across different vocabularies.

Abstract

Despite their strong performance, Dense Passage Retrieval (DPR) models suffer from a lack of interpretability. In this work, we propose a novel interpretability framework that leverages Sparse Autoencoders (SAEs) to decompose previously uninterpretable dense embeddings from DPR models into distinct, interpretable latent concepts. We generate natural language descriptions for each latent concept, enabling human interpretations of both the dense embeddings and the query-document similarity scores of DPR models. We further introduce Concept-Level Sparse Retrieval (CL-SR), a retrieval framework that directly utilizes the extracted latent concepts as indexing units. CL-SR effectively combines the semantic expressiveness of dense embeddings with the transparency and efficiency of sparse representations. We show that CL-SR achieves high index-space and computational efficiency while maintaining robust performance across vocabulary and semantic mismatches.