Scenario-Wise Rec: A Multi-Scenario Recommendation Benchmark

TL;DR

This paper introduces Scenario-Wise Rec, a comprehensive benchmark with datasets, models, and pipelines for multi-scenario recommendation, addressing current challenges in dataset processing and model transparency.

Contribution

The paper provides the first unified benchmark for MSR, including datasets, models, and evaluation pipelines, to facilitate fair comparisons and foster collaborative research.

Findings

Benchmark validated with industrial advertising data

Includes 6 public datasets and 12 models

Provides open-source code for reproducibility

Abstract

Multi Scenario Recommendation (MSR) tasks, referring to building a unified model to enhance performance across all recommendation scenarios, have recently gained much attention. However, current research in MSR faces two significant challenges that hinder the field's development: the absence of uniform procedures for multi-scenario dataset processing, thus hindering fair comparisons, and most models being closed-sourced, which complicates comparisons with current SOTA models. Consequently, we introduce our benchmark, \textbf{Scenario-Wise Rec}, which comprises 6 public datasets and 12 benchmark models, along with a training and evaluation pipeline. Additionally, we validated the benchmark using an industrial advertising dataset, reinforcing its reliability and applicability in real-world scenarios. We aim for this benchmark to offer researchers valuable insights from prior work, enabling the development of novel models based on our benchmark and thereby fostering a collaborative research ecosystem in MSR. Our source code is also publicly available.