Sample-Rank: Weak Multi-Objective Recommendations Using Rejection Sampling

TL;DR

This paper presents Sample-Rank, a scalable method for multi-objective recommendation that uses rejection sampling to align recommendations with marketplace goals, improving revenue without sacrificing conversion rates.

Contribution

Introduces a novel multi-goal sampling and ranking approach that simplifies multi-objective recommendation optimization for large-scale systems.

Findings

Achieved a 2.64% increase in revenue per order in online A/B tests.

Maintained conversion rates while biasing recommendations towards multiple objectives.

Reduced model development and deployment time in multi-objective settings.

Abstract

Online food ordering marketplaces are multi-stakeholder systems where recommendations impact the experience and growth of each participant in the system. A recommender system in this setting has to encapsulate the objectives and constraints of different stakeholders in order to find utility of an item for recommendation. Constrained-optimization based approaches to this problem typically involve complex formulations and have high computational complexity in production settings involving millions of entities. Simplifications and relaxation techniques (for example, scalarization) help but introduce sub-optimality and can be time-consuming due to the amount of tuning needed. In this paper, we introduce a method involving multi-goal sampling followed by ranking for user-relevance (Sample-Rank), to nudge recommendations towards multi-objective (MO) goals of the marketplace. The proposed method's novelty is that it reduces the MO recommendation problem to sampling from a desired multi-goal distribution then using it to build a production-friendly learning-to-rank (LTR) model. In offline experiments we show that we are able to bias recommendations towards MO criteria with acceptable trade-offs in metrics like AUC and NDCG. We also show results from a large-scale online A/B experiment where this approach gave a statistically significant lift of 2.64% in average revenue per order (RPO) (objective #1) with no drop in conversion rate (CR) (objective #2) while holding the average last-mile traversed flat (objective #3), vs. the baseline ranking method. This method also significantly reduces time to model development and deployment in MO settings and allows for trivial extensions to more objectives and other types of LTR models.