An IPW-based Unbiased Ranking Metric in Two-sided Markets

TL;DR

This paper introduces a novel unbiased ranking metric tailored for two-sided markets, extending IPW techniques to handle biases from both user groups, and demonstrates its effectiveness through real-world experiments.

Contribution

We develop a two-sided IPW estimator that accounts for biases from both user groups in two-sided markets, ensuring unbiasedness and improved ranking performance.

Findings

Outperforms baseline methods in real-world two-sided platforms

Effective in handling rare items with limited data

Ensures unbiasedness for the true ranking metric

Abstract

In modern recommendation systems, unbiased learning-to-rank (LTR) is crucial for prioritizing items from biased implicit user feedback, such as click data. Several techniques, such as Inverse Propensity Weighting (IPW), have been proposed for single-sided markets. However, less attention has been paid to two-sided markets, such as job platforms or dating services, where successful conversions require matching preferences from both users. This paper addresses the complex interaction of biases between users in two-sided markets and proposes a tailored LTR approach. We first present a formulation of feedback mechanisms in two-sided matching platforms and point out that their implicit feedback may include position bias from both user groups. On the basis of this observation, we extend the IPW estimator and propose a new estimator, named two-sided IPW, to address the position bases in two-sided markets. We prove that the proposed estimator satisfies the unbiasedness for the ground-truth ranking metric. We conducted numerical experiments on real-world two-sided platforms and demonstrated the effectiveness of our proposed method in terms of both precision and robustness. Our experiments showed that our method outperformed baselines especially when handling rare items, which are less frequently observed in the training data.