Self-Sampling Training and Evaluation for the Accuracy-Bias Tradeoff in Recommendation

TL;DR

This paper introduces a novel self-sampling training and evaluation framework to balance accuracy and bias in recommendation systems, addressing practical challenges in industrial applications without requiring specific data or architectures.

Contribution

The proposed SSTE framework enables effective bias management in recommendation systems through self-sampling, self-training, and self-evaluation modules, adaptable to real-world industrial settings.

Findings

SSTE effectively balances accuracy and bias in offline experiments.

SSTE improves recommendation quality in online A/B testing.

Framework is applicable without specialized data or models.

Abstract

Research on debiased recommendation has shown promising results. However, some issues still need to be handled for its application in industrial recommendation. For example, most of the existing methods require some specific data, architectures and training methods. In this paper, we first argue through an online study that arbitrarily removing all the biases in industrial recommendation may not consistently yield a desired performance improvement. For the situation that a randomized dataset is not available, we propose a novel self-sampling training and evaluation (SSTE) framework to achieve the accuracy-bias tradeoff in recommendation, i.e., eliminate the harmful biases and preserve the beneficial ones. Specifically, SSTE uses a self-sampling module to generate some subsets with different degrees of bias from the original training and validation data. A self-training module infers the beneficial biases and learns better tradeoff based on these subsets, and a self-evaluation module aims to use these subsets to construct more plausible references to reflect the optimized model. Finally, we conduct extensive offline experiments on two datasets to verify the effectiveness of our SSTE. Moreover, we deploy our SSTE in homepage recommendation of a famous financial management product called Tencent Licaitong, and find very promising results in an online A/B test.