How Researchers Could Obtain Quick and Cheap User Feedback on their Algorithms Without Having to Operate their Own Recommender System

TL;DR

This paper introduces a new evaluation tool that allows researchers to gather user feedback on recommendation algorithms without needing their own operational recommender system, thus lowering barriers for user-centric evaluation.

Contribution

The paper presents a novel evaluation paradigm and a tool enabling user-centric assessment of recommendation algorithms without access to an operational system.

Findings

Prototype of the evaluation tool developed

Initial experiments planned to validate the approach

Reduces cost and complexity of user-centric evaluation

Abstract

The majority of recommendation algorithms are evaluated on the basis of historic benchmark datasets. Evaluation on historic benchmark datasets is quick and cheap to conduct, yet excludes the viewpoint of users who actually consume recommendations. User feedback is seldom collected, since it requires access to an operational recommender system. Establishing and maintaining an operational recommender system imposes a timely and financial burden that a majority of researchers cannot shoulder. We aim to reduce this burden in order to promote widespread user-centric evaluations of recommendation algorithms, in particular for novice researchers in the field. We present work in progress on an evaluation tool that implements a novel paradigm that enables user-centric evaluations of recommendation algorithms without access to an operational recommender system. Finally, we sketch the experiments we plan to conduct with the help of the evaluation tool.