SSE: A Metric for Evaluating Search System Explainability

TL;DR

This paper introduces SSE, a new metric for evaluating the explainability of search systems, validated through user studies to distinguish explainable from non-explainable systems, advancing transparency in information retrieval.

Contribution

The paper presents SSE, a novel evaluation metric for search system explainability, and demonstrates its effectiveness through crowdsourced user studies.

Findings

SSE successfully differentiates explainable and non-explainable systems.

Higher SSE scores correlate with greater interpretability.

Non-native and native English speakers perceive similar temporal demand and performance.

Abstract

Explainable Information Retrieval (XIR) is a growing research area focused on enhancing transparency and trustworthiness of the complex decision-making processes taking place in modern information retrieval systems. While there has been progress in developing XIR systems, empirical evaluation tools to assess the degree of explainability attained by such systems are lacking. To close this gap and gain insights into the true merit of XIR systems, we extend existing insights from a factor analysis of search explainability to introduce SSE (Search System Explainability), an evaluation metric for XIR search systems. Through a crowdsourced user study, we demonstrate SSE's ability to distinguish between explainable and non-explainable systems, showing that systems with higher scores indeed indicate greater interpretability. Additionally, we observe comparable perceived temporal demand and performance levels between non-native and native English speakers. We hope that aside from these concrete contributions to XIR, this line of work will serve as a blueprint for similar explainability evaluation efforts in other domains of machine learning and natural language processing.