Interpreting Multilingual and Document-Length Sensitive Relevance Computations in Neural Retrieval Models through Axiomatic Causal Interventions

TL;DR

This study reproduces and extends previous work on neural retrieval models, demonstrating that they encode term frequency and document length information across languages, with implications for interpretability and reproducibility in IR models.

Contribution

It confirms the encoding of term frequency in neural models across languages and extends analysis to document length, using activation patching for interpretability.

Findings

Term frequency encoding generalizes across languages.

Activation patching isolates model components responsible for relevance.

Later layers encode sequence-level information in CLS tokens.

Abstract

This reproducibility study analyzes and extends the paper "Axiomatic Causal Interventions for Reverse Engineering Relevance Computation in Neural Retrieval Models," which investigates how neural retrieval models encode task-relevant properties such as term frequency. We reproduce key experiments from the original paper, confirming that information on query terms is captured in the model encoding. We extend this work by applying activation patching to Spanish and Chinese datasets and by exploring whether document-length information is encoded in the model as well. Our results confirm that the designed activation patching method can isolate the behavior to specific components and tokens in neural retrieval models. Moreover, our findings indicate that the location of term frequency generalizes across languages and that in later layers, the information for sequence-level tasks is represented in the CLS token. The results highlight the need for further research into interpretability in information retrieval and reproducibility in machine learning research. Our code is available at https://github.com/OliverSavolainen/axiomatic-ir-reproduce.