U-rank: Utility-oriented Learning to Rank with Implicit Feedback

TL;DR

U-rank is a novel learning-to-rank framework that directly optimizes utility using implicit feedback, incorporating position-aware click prediction and attention bias modeling, leading to significant improvements in web search and recommender systems.

Contribution

The paper introduces U-rank, a new utility-oriented ranking framework that optimizes expected utility directly and models attention bias at item and query levels.

Findings

U-rank outperforms state-of-the-art methods on benchmark datasets.

U-rank achieves large online performance improvements in commercial recommender systems.

The Lambdaloss optimization framework is both theoretically sound and empirically effective.

Abstract

Learning to rank with implicit feedback is one of the most important tasks in many real-world information systems where the objective is some specific utility, e.g., clicks and revenue. However, we point out that existing methods based on probabilistic ranking principle do not necessarily achieve the highest utility. To this end, we propose a novel ranking framework called U-rank that directly optimizes the expected utility of the ranking list. With a position-aware deep click-through rate prediction model, we address the attention bias considering both query-level and item-level features. Due to the item-specific attention bias modeling, the optimization for expected utility corresponds to a maximum weight matching on the item-position bipartite graph. We base the optimization of this objective in an efficient Lambdaloss framework, which is supported by both theoretical and empirical analysis. We conduct extensive experiments for both web search and recommender systems over three benchmark datasets and two proprietary datasets, where the performance gain of U-rank over state-of-the-arts is demonstrated. Moreover, our proposed U-rank has been deployed on a large-scale commercial recommender and a large improvement over the production baseline has been observed in an online A/B testing.