Offline Evaluation of Reward-Optimizing Recommender Systems: The Case of Simulation

TL;DR

This paper advocates for simulation-based offline evaluation methods for reward-optimizing recommender systems, highlighting their advantages over proxy and counterfactual metrics in providing more reliable assessments.

Contribution

It introduces simulation as a promising alternative for offline evaluation, addressing limitations of existing proxy and counterfactual metrics in real-world environments.

Findings

Simulation-based comparisons offer more reliable evaluation than traditional offline metrics.

Offline metrics like proxy and counterfactual methods often lack correlation with online performance.

Simulation can bridge the gap between offline evaluation and real-world system performance.

Abstract

Both in academic and industry-based research, online evaluation methods are seen as the golden standard for interactive applications like recommendation systems. Naturally, the reason for this is that we can directly measure utility metrics that rely on interventions, being the recommendations that are being shown to users. Nevertheless, online evaluation methods are costly for a number of reasons, and a clear need remains for reliable offline evaluation procedures. In industry, offline metrics are often used as a first-line evaluation to generate promising candidate models to evaluate online. In academic work, limited access to online systems makes offline metrics the de facto approach to validating novel methods. Two classes of offline metrics exist: proxy-based methods, and counterfactual methods. The first class is often poorly correlated with the online metrics we care about, and the latter class only provides theoretical guarantees under assumptions that cannot be fulfilled in real-world environments. Here, we make the case that simulation-based comparisons provide ways forward beyond offline metrics, and argue that they are a preferable means of evaluation.