Ranking metrics on non-shuffled traffic

TL;DR

This paper proposes a novel method to address position bias in ranking metrics by leveraging the stochasticity of the recommendation policy, avoiding the need for uniform shuffling which can harm user experience.

Contribution

It introduces a new approach that uses the inherent stochasticity of recommendation policies to correct position bias in ranking metrics.

Findings

The proposed method effectively reduces position bias in non-shuffled traffic.

It maintains user experience while improving metric accuracy.

The approach is applicable to real-world recommender systems.

Abstract

Ranking metrics are a family of metrics largely used to evaluate recommender systems. However they typically suffer from the fact the reward is affected by the order in which recommended items are displayed to the user. A classical way to overcome this position bias is to uniformly shuffle a proportion of the recommendations, but this method may result in a bad user experience. It is nevertheless common to use a stochastic policy to generate the recommendations, and we suggest a new method to overcome the position bias, by leveraging the stochasticity of the policy used to collect the dataset.