Measuring Fairness in Large-Scale Recommendation Systems with Missing Labels

TL;DR

This paper introduces a novel method using randomized traffic to accurately measure fairness in large-scale recommendation systems with missing labels, supported by theoretical bounds and empirical validation on real data.

Contribution

It is the first to highlight the importance of random traffic in fairness measurement, propose a new estimation method, and provide a fairness dataset from TikTok.

Findings

The proposed method accurately estimates fairness metrics with theoretical error bounds.

Empirical results validate the effectiveness of the method on synthetic and real TikTok data.

The study emphasizes the importance of randomized data collection for fairness in recommendation systems.

Abstract

In large-scale recommendation systems, the vast array of items makes it infeasible to obtain accurate user preferences for each product, resulting in a common issue of missing labels. Typically, only items previously recommended to users have associated ground truth data. Although there is extensive research on fairness concerning fully observed user-item interactions, the challenge of fairness in scenarios with missing labels remains underexplored. Previous methods often treat these samples missing labels as negative, which can significantly deviate from the ground truth fairness metrics. Our study addresses this gap by proposing a novel method employing a small randomized traffic to estimate fairness metrics accurately. We present theoretical bounds for the estimation error of our fairness metric and support our findings with empirical evidence on real data. Our numerical experiments on synthetic and TikTok's real-world data validate our theory and show the efficiency and effectiveness of our novel methods. To the best of our knowledge, we are the first to emphasize the necessity of random traffic in dataset collection for recommendation fairness, the first to publish a fairness-related dataset from TikTok and to provide reliable estimates of fairness metrics in the context of large-scale recommendation systems with missing labels.