User Model-Based Intent-Aware Metrics for Multilingual Search Evaluation

TL;DR

This paper introduces a new intent-aware metric for multilingual search evaluation that accounts for user language preferences and improves correlation with online satisfaction metrics.

Contribution

It develops a novel user behavior model for multilingual search, addressing limitations of existing models and enhancing offline evaluation accuracy.

Findings

New intent-aware metric correlates better with online satisfaction

Proposed user behavior model overcomes previous limitations

Improved offline evaluation for multilingual search quality

Abstract

Despite the growing importance of multilingual aspect of web search, no appropriate offline metrics to evaluate its quality are proposed so far. At the same time, personal language preferences can be regarded as intents of a query. This approach translates the multilingual search problem into a particular task of search diversification. Furthermore, the standard intent-aware approach could be adopted to build a diversified metric for multilingual search on the basis of a classical IR metric such as ERR. The intent-aware approach estimates user satisfaction under a user behavior model. We show however that the underlying user behavior models is not realistic in the multilingual case, and the produced intent-aware metric do not appropriately estimate the user satisfaction. We develop a novel approach to build intent-aware user behavior models, which overcome these limitations and convert to quality metrics that better correlate with standard online metrics of user satisfaction.