An Empirical Analysis on Transparent Algorithmic Exploration in Recommender Systems

TL;DR

This paper introduces a transparent recommendation interface that improves user feedback and satisfaction by clearly indicating exploratory items, outperforming traditional mix-in methods in user trust, diversity, and engagement.

Contribution

The study proposes a novel transparent interface for exploration in recommender systems, demonstrating its effectiveness over traditional mix-in approaches through empirical user study results.

Findings

Users provided more feedback on exploratory items with the new interface.

The new interface improved user perceptions of novelty, diversity, trust, and satisfaction.

Exploration increased user-centric evaluation metrics only with the transparent interface.

Abstract

All learning algorithms for recommendations face inevitable and critical trade-off between exploiting partial knowledge of a user's preferences for short-term satisfaction and exploring additional user preferences for long-term coverage. Although exploration is indispensable for long success of a recommender system, the exploration has been considered as the risk to decrease user satisfaction. The reason for the risk is that items chosen for exploration frequently mismatch with the user's interests. To mitigate this risk, recommender systems have mixed items chosen for exploration into a recommendation list, disguising the items as recommendations to elicit feedback on the items to discover the user's additional tastes. This mix-in approach has been widely used in many recommenders, but there is rare research, evaluating the effectiveness of the mix-in approach or proposing a new approach for eliciting user feedback without deceiving users. In this work, we aim to propose a new approach for feedback elicitation without any deception and compare our approach to the conventional mix-in approach for evaluation. To this end, we designed a recommender interface that reveals which items are for exploration and conducted a within-subject study with 94 MTurk workers. Our results indicated that users left significantly more feedback on items chosen for exploration with our interface. Besides, users evaluated that our new interface is better than the conventional mix-in interface in terms of novelty, diversity, transparency, trust, and satisfaction. Finally, path analysis show that, in only our new interface, exploration caused to increase user-centric evaluation metrics. Our work paves the way for how to design an interface, which utilizes learning algorithm based on users' feedback signals, giving better user experience and gathering more feedback data.