On (Normalised) Discounted Cumulative Gain as an Off-Policy Evaluation Metric for Top-$n$ Recommendation

TL;DR

This paper critically examines the use of (normalized) Discounted Cumulative Gain (nDCG) as an offline metric for top-$n$ recommendation evaluation, revealing limitations and conditions for its reliability compared to online experiments.

Contribution

It formally analyzes the assumptions behind DCG as an unbiased estimator of online reward and demonstrates that normalizing DCG can lead to inconsistent rankings of methods.

Findings

Unnormalized DCG correlates well with online reward under certain assumptions.

Normalizing DCG can invert the ranking of methods, reducing its reliability.

Empirical analysis shows unnormalized DCG's practical utility in offline evaluation.

Abstract

Approaches to recommendation are typically evaluated in one of two ways: (1) via a (simulated) online experiment, often seen as the gold standard, or (2) via some offline evaluation procedure, where the goal is to approximate the outcome of an online experiment. Several offline evaluation metrics have been adopted in the literature, inspired by ranking metrics prevalent in the field of Information Retrieval. (Normalised) Discounted Cumulative Gain (nDCG) is one such metric that has seen widespread adoption in empirical studies, and higher (n)DCG values have been used to present new methods as the state-of-the-art in top-$n$ recommendation for many years. Our work takes a critical look at this approach, and investigates when we can expect such metrics to approximate the gold standard outcome of an online experiment. We formally present the assumptions that are necessary to consider DCG an unbiased estimator of online reward and provide a derivation for this metric from first principles, highlighting where we deviate from its traditional uses in IR. Importantly, we show that normalising the metric renders it inconsistent, in that even when DCG is unbiased, ranking competing methods by their normalised DCG can invert their relative order. Through a correlation analysis between off- and on-line experiments conducted on a large-scale recommendation platform, we show that our unbiased DCG estimates strongly correlate with online reward, even when some of the metric's inherent assumptions are violated. This statement no longer holds for its normalised variant, suggesting that nDCG's practical utility may be limited.