Towards a Signal Detection Based Measure for Assessing Information Quality of Explainable Recommender Systems

TL;DR

This paper introduces an objective, signal detection theory-based metric to evaluate the information quality of explanations in recommender systems, focusing on Veracity through Fidelity and Attunement.

Contribution

It proposes a novel metric for assessing explanation quality in recommender systems, addressing the lack of objective measures for Veracity.

Findings

The metric effectively distinguishes different levels of explanation quality.

It provides meaningful insights into explanation Veracity.

The approach offers an objective alternative to subjective user studies.

Abstract

There is growing interest in explainable recommender systems that provide recommendations along with explanations for the reasoning behind them. When evaluating recommender systems, most studies focus on overall recommendation performance. Only a few assess the quality of the explanations. Explanation quality is often evaluated through user studies that subjectively gather users' opinions on representative explanatory factors that shape end-users' perspective towards the results, not about the explanation contents itself. We aim to fill this gap by developing an objective metric to evaluate Veracity: the information quality of explanations. Specifically, we decompose Veracity into two dimensions: Fidelity and Attunement. Fidelity refers to whether the explanation includes accurate information about the recommended item. Attunement evaluates whether the explanation reflects the target user's preferences. By applying signal detection theory, we first determine decision outcomes for each dimension and then combine them to calculate a sensitivity, which serves as the final Veracity value. To assess the effectiveness of the proposed metric, we set up four cases with varying levels of information quality to validate whether our metric can accurately capture differences in quality. The results provided meaningful insights into the effectiveness of our proposed metric.