Diversification as Risk Minimization

TL;DR

This paper introduces VRisk and VRisker, a risk-minimization framework for diversification in search ranking, improving robustness against minority user intents with minimal impact on average relevance.

Contribution

It formulates diversification as a risk minimization problem, proposes VRisk as a new metric, and develops VRisker, a greedy algorithm with provable guarantees for robust ranking.

Findings

VRisker reduces worst-case intent failures by up to 33%.

Existing diversification methods are not more robust than naive approaches.

Experiments on multiple datasets validate the effectiveness of VRisker.

Abstract

Users tend to remember failures of a search session more than its many successes. This observation has led to work on search robustness, where systems are penalized if they perform very poorly on some queries. However, this principle of robustness has been overlooked within a single query. An ambiguous or underspecified query (e.g., ``jaguar'') can have several user intents, where popular intents often dominate the ranking, leaving users with minority intents unsatisfied. Although the diversification literature has long recognized this issue, existing metrics only model the average relevance across intents and provide no robustness guarantees. More surprisingly, we show theoretically and empirically that many well-known diversification algorithms are no more robust than a naive, non-diversified algorithm. To address this critical gap, we propose to frame diversification as a risk-minimization problem. We introduce VRisk, which measures the expected risk faced by the least-served fraction of intents in a query. Optimizing VRisk produces a robust ranking, reducing the likelihood of poor user experiences. We then propose VRisker, a fast greedy re-ranker with provable approximation guarantees. Finally, experiments on NTCIR INTENT-2, TREC Web 2012, and MovieLens show the vulnerability of existing methods. VRisker reduces worst-case intent failures by up to 33% with a minimal 2% drop in average performance.