Measuring the Business Value of Recommender Systems

TL;DR

This paper reviews how recommender systems create business value in real-world settings, analyzing existing studies and highlighting challenges in measuring their economic impact.

Contribution

It provides a comprehensive review of field tests of recommender systems, discussing challenges and gaps in measuring business value and algorithm performance.

Findings

Field tests use diverse performance measures

Measuring real-world business impact remains challenging

Academic evaluations often lack practical relevance

Abstract

Recommender Systems are nowadays successfully used by all major web sites (from e-commerce to social media) to filter content and make suggestions in a personalized way. Academic research largely focuses on the value of recommenders for consumers, e.g., in terms of reduced information overload. To what extent and in which ways recommender systems create business value is, however, much less clear, and the literature on the topic is scattered. In this research commentary, we review existing publications on field tests of recommender systems and report which business-related performance measures were used in such real-world deployments. We summarize common challenges of measuring the business value in practice and critically discuss the value of algorithmic improvements and offline experiments as commonly done in academic environments. Overall, our review indicates that various open questions remain both regarding the realistic quantification of the business effects of recommenders and the performance assessment of recommendation algorithms in academia.