Understanding Wacky Weights: A Dissection of SPLADE's Learned Term Importance

TL;DR

This paper systematically investigates the phenomenon of wacky weights in SPLADE models, revealing their origins, prevalence, and impact on retrieval effectiveness through comprehensive experiments and formal analysis.

Contribution

It provides the first formal definition of wacky weights, introduces a novel measure for their prevalence, and analyzes factors influencing their occurrence in SPLADE models.

Findings

Larger vocabularies increase wacky token prevalence.

Stricter sparsity regularizers reduce wacky token prevalence.

Wacky weights mainly contribute to in-domain retrieval effectiveness.

Abstract

Learned sparse retrieval models such as SPLADE combine the effectiveness of neural architectures with the efficiency of inverted indices. As these models assign weights to terms from a fixed vocabulary, interpretability is often touted as a major benefit of these models. However, the emergence of wacky weights, i.e., expansion terms that appear semantically unrelated to the input, limits interpretability. While prior research has anecdotally observed this phenomenon, there is a lack of systematic understanding regarding their origins, prevalence, and contribution to retrieval effectiveness. In this paper, we reproduce SPLADE-v2 to systematically investigate wacky weights across the SPLADE family of models. We present a comprehensive dissection of wacky weights, providing a formal definition of wackiness based on the lexical utility of expansion terms. Furthermore, we introduce a novel measure to compare the prevalence of these tokens across models with varying vocabularies and sparsity levels. Beyond reproducing the original SPLADE-v2, we train it with various loss functions, datasets, and backbone transformers to isolate the factors contributing to wackiness. Our results show that larger vocabularies are associated with a higher prevalence of wacky tokens, while stricter sparsity regularizers are associated with lower prevalence. Finally, we find that wacky weights are used primarily for in-domain effectiveness rather than out-of-domain generalization.