Helping results assessment by adding explainable elements to the deep relevance matching model

TL;DR

This paper introduces explainable visual elements for search engine results, including query term weights and passage relevance, to improve user trust and experience, validated through user studies.

Contribution

It proposes novel visualizations for query term importance and passage relevance in search results, enhancing explainability and user trust.

Findings

Users find the explainable interface more understandable and assessable.

Explainability improves user trust but does not significantly increase relevant document selection.

Proposed visualizations are promising for enhancing search engine transparency.

Abstract

In this paper we address the explainability of web search engines. We propose two explainable elements on the search engine result page: a visualization of query term weights and a visualization of passage relevance. The idea is that search engines that indicate to the user why results are retrieved are valued higher by users and gain user trust. We deduce the query term weights from the term gating network in the Deep Relevance Matching Model (DRMM) and visualize them as a doughnut chart. In addition, we train a passage-level ranker with DRMM that selects the most relevant passage from each document and shows it as snippet on the result page. Next to the snippet we show a document thumbnail with this passage highlighted. We evaluate the proposed interface in an online user study, asking users to judge the explainability and assessability of the interface. We found that users judge our proposed interface significantly more explainable and easier to assess than a regular search engine result page. However, they are not significantly better in selecting the relevant documents from the top-5. This indicates that the explainability of the search engine result page leads to a better user experience. Thus, we conclude that the proposed explainable elements are promising as visualization for search engine users.