Towards Group-aware Search Success

TL;DR

This paper introduces a new group-aware metric for search success that considers demographic diversity, along with a ranking model to better align search results with varied user needs, validated through real-world datasets.

Contribution

It proposes the GA-SS metric and gMPC ranking model, addressing demographic disparities in search success measurement and improving inclusivity in search evaluation.

Findings

GA-SS effectively captures demographic satisfaction differences.

Stochastic ranking policies influence search success metrics.

The approach enhances understanding of search quality across user groups.

Abstract

Traditional measures of search success often overlook the varying information needs of different demographic groups. To address this gap, we introduce a novel metric, named Group-aware Search Success (GA-SS). GA-SS redefines search success to ensure that all demographic groups achieve satisfaction from search outcomes. We introduce a comprehensive mathematical framework to calculate GA-SS, incorporating both static and stochastic ranking policies and integrating user browsing models for a more accurate assessment. In addition, we have proposed Group-aware Most Popular Completion (gMPC) ranking model to account for demographic variances in user intent, aligning more closely with the diverse needs of all user groups. We empirically validate our metric and approach with two real-world datasets: one focusing on query auto-completion and the other on movie recommendations, where the results highlight the impact of stochasticity and the complex interplay among various search success metrics. Our findings advocate for a more inclusive approach in measuring search success, as well as inspiring future investigations into the quality of service of search.