A Counterfactual Explanation Framework for Retrieval Models

TL;DR

This paper introduces a counterfactual explanation framework for retrieval models, focusing on identifying which terms could improve a document's ranking and explaining why it was not favored.

Contribution

It presents the first counterfactual approach to understand why documents are not ranked highly, applicable to both statistical and deep-learning retrieval models.

Findings

Effective in predicting counterfactuals for BM25, DRMM, DSSM, ColBERT, MonoT5.

Addresses the gap in explainability by focusing on non-favored documents.

Enhances transparency in retrieval model decisions.

Abstract

Explainability has become a crucial concern in today's world, aiming to enhance transparency in machine learning and deep learning models. Information retrieval is no exception to this trend. In existing literature on explainability of information retrieval, the emphasis has predominantly been on illustrating the concept of relevance concerning a retrieval model. The questions addressed include why a document is relevant to a query, why one document exhibits higher relevance than another, or why a specific set of documents is deemed relevant for a query. However, limited attention has been given to understanding why a particular document is not favored (e.g., not within top-K) with respect to a query and a retrieval model. In an effort to address this gap, our work focuses on the question of what terms need to be added within a document to improve its ranking. This, in turn, answers the question of which words in the document played a role in not being favored by a retrieval model for a particular query. We use a counterfactual framework to solve the above-mentioned research problem. % To the best of our knowledge, we mark the first attempt to tackle this specific counterfactual problem (i.e. examining the absence of which words can affect the ranking of a document). Our experiments show the effectiveness of our proposed approach in predicting counterfactuals for both statistical (e.g. BM25) and deep-learning-based models (e.g. DRMM, DSSM, ColBERT, MonoT5).