E Pluribus Unum: Guidelines on Multi-Objective Evaluation of Recommender Systems

TL;DR

This paper discusses the importance of multi-objective evaluation in recommender systems, highlighting insights from a data challenge and proposing guidelines for balanced, comprehensive assessment beyond accuracy.

Contribution

It introduces a first-principles approach to multi-objective model selection and provides guidelines for conducting evaluation challenges in recommender systems.

Findings

Insights from EvalRS 2022 on balancing multiple objectives

Formulation of a first-principles approach to multi-objective evaluation

Guidelines for multi-objective evaluation challenges

Abstract

Recommender Systems today are still mostly evaluated in terms of accuracy, with other aspects beyond the immediate relevance of recommendations, such as diversity, long-term user retention and fairness, often taking a back seat. Moreover, reconciling multiple performance perspectives is by definition indeterminate, presenting a stumbling block to those in the pursuit of rounded evaluation of Recommender Systems. EvalRS 2022 -- a data challenge designed around Multi-Objective Evaluation -- was a first practical endeavour, providing many insights into the requirements and challenges of balancing multiple objectives in evaluation. In this work, we reflect on EvalRS 2022 and expound upon crucial learnings to formulate a first-principles approach toward Multi-Objective model selection, and outline a set of guidelines for carrying out a Multi-Objective Evaluation challenge, with potential applicability to the problem of rounded evaluation of competing models in real-world deployments.