Whom do Explanations Serve? A Systematic Literature Survey of User Characteristics in Explainable Recommender Systems Evaluation

TL;DR

This paper systematically reviews 124 studies on explainable recommender systems, highlighting the limited and inconsistent consideration of user characteristics, which affects the generalizability and reproducibility of findings.

Contribution

It provides a comprehensive survey of user studies in explainable recommender systems and identifies gaps in participant diversity and data reporting practices.

Findings

Most studies focus on non-representative user groups

Inconsistent data reporting hampers reproducibility

Limited consideration of user characteristics in evaluations

Abstract

Adding explanations to recommender systems is said to have multiple benefits, such as increasing user trust or system transparency. Previous work from other application areas suggests that specific user characteristics impact the users' perception of the explanation. However, we rarely find this type of evaluation for recommender systems explanations. This paper addresses this gap by surveying 124 papers in which recommender systems explanations were evaluated in user studies. We analyzed their participant descriptions and study results where the impact of user characteristics on the explanation effects was measured. Our findings suggest that the results from the surveyed studies predominantly cover specific users who do not necessarily represent the users of recommender systems in the evaluation domain. This may seriously hamper the generalizability of any insights we may gain from current studies on explanations in recommender systems. We further find inconsistencies in the data reporting, which impacts the reproducibility of the reported results. Hence, we recommend actions to move toward a more inclusive and reproducible evaluation.