Deep Uncertainty-Based Explore for Index Construction and Retrieval in Recommendation System

TL;DR

This paper introduces UICR, a novel uncertainty-based method for index construction and retrieval in recommendation systems, enhancing novelty while maintaining relevance through uncertainty modeling.

Contribution

The paper proposes a new uncertainty modeling approach in matching algorithms, improving the balance of relevance and novelty in recommendation systems.

Findings

UICR improves recommendation novelty without reducing relevance.

Experimental results show UICR's effectiveness on real-world and open-source datasets.

Online A/B testing confirms UICR's performance in industrial environments.

Abstract

In recommendation systems, the relevance and novelty of the final results are selected through a cascade system of Matching -> Ranking -> Strategy. The matching model serves as the starting point of the pipeline and determines the upper bound of the subsequent stages. Balancing the relevance and novelty of matching results is a crucial step in the design and optimization of recommendation systems, contributing significantly to improving recommendation quality. However, the typical matching algorithms have not simultaneously addressed the relevance and novelty perfectly. One main reason is that deep matching algorithms exhibit significant uncertainty when estimating items in the long tail (e.g., due to insufficient training samples) items.The uncertainty not only affects the training of the models but also influences the confidence in the index construction and beam search retrieval process of these models. This paper proposes the UICR (Uncertainty-based explore for Index Construction and Retrieval) algorithm, which introduces the concept of uncertainty modeling in the matching stage and achieves multi-task modeling of model uncertainty and index uncertainty. The final matching results are obtained by combining the relevance score and uncertainty score infered by the model. Experimental results demonstrate that the UICR improves novelty without sacrificing relevance on realworld industrial productive environments and multiple open-source datasets. Remarkably, online A/B test results of display advertising in Shopee demonstrates the effectiveness of the proposed algorithm.