Spectral Tempering for Embedding Compression in Dense Passage Retrieval

TL;DR

Spectral Tempering (SpecTemp) is a learning-free, adaptive spectral scaling method for embedding compression in dense passage retrieval, outperforming fixed hyperparameter approaches without requiring labeled data.

Contribution

We introduce Spectral Tempering, an adaptive spectral scaling technique that automatically determines optimal scaling based on eigenspectrum analysis, eliminating the need for hyperparameter tuning.

Findings

SpecTemp achieves near-oracle performance compared to grid search.

It is fully learning-free and model-agnostic.

Extensive experiments validate its effectiveness across tasks.

Abstract

Dimensionality reduction is critical for deploying dense retrieval systems at scale, yet mainstream post-hoc methods face a fundamental trade-off: principal component analysis (PCA) preserves dominant variance but underutilizes representational capacity, while whitening enforces isotropy at the cost of amplifying noise in the heavy-tailed eigenspectrum of retrieval embeddings. Intermediate spectral scaling methods unify these extremes by reweighting dimensions with a power coefficient $\gamma$, but treat $\gamma$ as a fixed hyperparameter that requires task-specific tuning. We show that the optimal scaling strength $\gamma$ is not a global constant: it varies systematically with target dimensionality $k$ and is governed by the signal-to-noise ratio (SNR) of the retained subspace. Based on this insight, we propose Spectral Tempering (\textbf{SpecTemp}), a learning-free method that derives an adaptive $\gamma(k)$ directly from the corpus eigenspectrum using local SNR analysis and knee-point normalization, requiring no labeled data or validation-based search. Extensive experiments demonstrate that Spectral Tempering consistently achieves near-oracle performance relative to grid-searched $\gamma^*(k)$ while remaining fully learning-free and model-agnostic. Our code is publicly available at https://github.com/liyongkang123/SpecTemp.