CROLoss: Towards a Customizable Loss for Retrieval Models in Recommender Systems

TL;DR

This paper introduces CROLoss, a customizable loss function that directly optimizes Recall@N for retrieval models in recommender systems, outperforming traditional methods and boosting online revenue.

Contribution

The paper proposes CROLoss, a novel loss function that can be tailored for different retrieval sizes N and directly optimize Recall@N, with a gradient-based Lambda method for enhanced performance.

Findings

CROLoss achieves state-of-the-art results on benchmark datasets.

CROLoss improves online revenue by 4.75% in real-world deployment.

The Lambda method enhances the flexibility and effectiveness of CROLoss.

Abstract

In large-scale recommender systems, retrieving top N relevant candidates accurately with resource constrain is crucial. To evaluate the performance of such retrieval models, Recall@N, the frequency of positive samples being retrieved in the top N ranking, is widely used. However, most of the conventional loss functions for retrieval models such as softmax cross-entropy and pairwise comparison methods do not directly optimize Recall@N. Moreover, those conventional loss functions cannot be customized for the specific retrieval size N required by each application and thus may lead to sub-optimal performance. In this paper, we proposed the Customizable Recall@N Optimization Loss (CROLoss), a loss function that can directly optimize the Recall@N metrics and is customizable for different choices of N. This proposed CROLoss formulation defines a more generalized loss function space, covering most of the conventional loss functions as special cases. Furthermore, we develop the Lambda method, a gradient-based method that invites more flexibility and can further boost the system performance. We evaluate the proposed CROLoss on two public benchmark datasets. The results show that CROLoss achieves SOTA results over conventional loss functions for both datasets with various choices of retrieval size N. CROLoss has been deployed onto our online E-commerce advertising platform, where a fourteen-day online A/B test demonstrated that CROLoss contributes to a significant business revenue growth of 4.75%.