Learning Retrieval Models with Sparse Autoencoders
Thibault Formal, Maxime Louis, Herv\'e Dejean, St\'ephane Clinchant
TL;DR
This paper introduces SPLARE, a novel method for training sparse autoencoder-based retrieval models that produce interpretable, language-agnostic representations, outperforming traditional vocabulary-based methods in multilingual and out-of-domain retrieval tasks.
Contribution
The paper presents SPLARE, a new approach leveraging sparse autoencoders for learned sparse retrieval, demonstrating improved performance and interpretability over existing vocabulary-based models.
Findings
SPLARE outperforms vocabulary-based LSR in multilingual settings.
SPLARE-7B achieves top results on multilingual and English retrieval tasks.
A lightweight 2B-parameter variant maintains strong performance.
Abstract
Sparse autoencoders (SAEs) provide a powerful mechanism for decomposing the dense representations produced by Large Language Models (LLMs) into interpretable latent features. We posit that SAEs constitute a natural foundation for Learned Sparse Retrieval (LSR), whose objective is to encode queries and documents into high-dimensional sparse representations optimized for efficient retrieval. In contrast to existing LSR approaches that project input sequences into the vocabulary space, SAE-based representations offer the potential to produce more semantically structured, expressive, and language-agnostic features. Building on this insight, we introduce SPLARE, a method to train SAE-based LSR models. Our experiments, relying on recently released open-source SAEs, demonstrate that this technique consistently outperforms vocabulary-based LSR in multilingual and out-of-domain settings.…
Peer Reviews
Decision·ICLR 2026 Poster
The main strengths of the paper are as follows: 1. Idea of using latent features for document and query representation might have a lot of applications in text retrieval 2. The SPLARE learned sparse retrieval framework is interesting. 3. Detailed performance comparison with SOTA text retrieval models.
The main Weaknesses of the paper are as follows: 1. Details about SPLARE working is missing. It might need some more explanation. 2. Terms like SAE width needs some more details. 3. Limited novels of the overall retrieval model when compared to sparse embed and other LSR techniques. 4. How SAEs are trained is not clear from the paper. 5. Paper writing have a lot of scope of improvement. 6. Limited performance improvement on MTEB retrieval benchmark when compared to SPLADE-LLama
1. The paper is the first to systematically use the pre-trained sparse autoencoder as the "implicit vocabulary" of LSR, replacing the traditional methods based on the original tokenizer vocabulary. 2. The proposed method demonstrates outstanding performance in multilingual and cross-domain scenarios.
1. Although the authors find that the best performance was achieved at Layer 26 for different SAE widths, this conclusion is only drawn from the experiments conducted on the Llama-3.1-8B model. If readers wish to apply the SPLARE method to other models (such as Gemma), there is no guarantee that the selection of this layer will be successful. It is suggested that the author provide more experimental results on the models and the layer selection strategies to enhance the generalization ability of
1.The paper originally combines the semantic feature decomposition of SAEs with learned sparse retrieval; 2.The experimental results are impressive. SPLARE consistently outperforms SPLADE-Llama in multilingual and out-of-domain settings and achieves a level of performance comparable to SOTA dense models on comprehensive benchmarks like MMTEB. 3.Excellent Generalization and Efficiency: SPLARE demonstrates strong multilingual generalization even when trained only on English data.
1. Lack of Detailed Cost Analysis: The introduction of the SAE module incurs additional computational and memory costs at inference time. While the paper mentions mitigating this by using intermediate LLM layers, it does not quantify the latency and memory overhead introduced by the SAE projection step itself. 2. Insufficient Context for Sparsity: The paper controls document vectors to ~400 non-zero dimensions. While this is sparse in a >130k-dimensional space, its advantage is not immediately
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Multimodal Machine Learning Applications · Information Retrieval and Search Behavior