Evaluating Performance and Bias of Negative Sampling in Large-Scale Sequential Recommendation Models

TL;DR

This paper compares various negative sampling methods for large-scale sequential recommendation models, analyzing their impact on performance and bias across different dataset characteristics and popularity biases.

Contribution

It provides a comprehensive empirical evaluation of negative sampling techniques, highlighting their effects on model performance and bias in large-scale recommendation systems.

Findings

Random sampling reinforces popularity bias and favors head items.

Popularity-based methods offer more balanced performance across popularity bands.

Choice of negative sampling method significantly impacts model bias and effectiveness.

Abstract

Large-scale industrial recommendation models predict the most relevant items from catalogs containing millions or billions of options. To train these models efficiently, a small set of irrelevant items (negative samples) is selected from the vast catalog for each relevant item (positive example), helping the model distinguish between relevant and irrelevant items. Choosing the right negative sampling method is a common challenge. We address this by implementing and comparing various negative sampling methods - random, popularity-based, in-batch, mixed, adaptive, and adaptive with mixed variants - on modern sequential recommendation models. Our experiments, including hyperparameter optimization and 20x repeats on three benchmark datasets with varying popularity biases, show how the choice of method and dataset characteristics impact key model performance metrics. We also reveal that average performance metrics often hide imbalances across popularity bands (head, mid, tail). We find that commonly used random negative sampling reinforces popularity bias and performs best for head items. Popularity-based methods (in-batch and global popularity negative sampling) can offer balanced performance at the cost of lower overall model performance results. Our study serves as a practical guide to the trade-offs in selecting a negative sampling method for large-scale sequential recommendation models. Code, datasets, experimental results and hyperparameters are available at: https://github.com/apple/ml-negative-sampling.