Which Neurons Matter in IR? Applying Integrated Gradients-based Methods to Understand Cross-Encoders

TL;DR

This paper adapts Integrated Gradients methods to interpret cross-encoder IR models, revealing the significance of certain neurons and validating their roles through pruning experiments.

Contribution

It introduces a novel application of Integrated Gradients for neuron importance analysis in IR models, focusing on relevance neurons and their behavior with unseen data.

Findings

Identification of key relevance neurons in IR models

Insights into neuron behavior with unseen data

Validation of neuron importance through pruning

Abstract

With the recent addition of Retrieval-Augmented Generation (RAG), the scope and importance of Information Retrieval (IR) has expanded. As a result, the importance of a deeper understanding of IR models also increases. However, interpretability in IR remains under-explored, especially when it comes to the models' inner mechanisms. In this paper, we explore the possibility of adapting Integrated Gradient-based methods in an IR context to identify the role of individual neurons within the model. In particular, we provide new insights into the role of what we call "relevance" neurons, as well as how they deal with unseen data. Finally, we carry out an in-depth pruning study to validate our findings.