Beyond Cumulated Gain and Average Precision: Including Willingness and Expectation in the User Model

TL;DR

This paper introduces a new metric family for information retrieval that incorporates user willingness and expectation into the evaluation process, aiming to better model user satisfaction and stopping criteria.

Contribution

It proposes a unified framework combining willingness and expectation to define a new set of metrics for user satisfaction in IR evaluation.

Findings

New metric family based on willingness and expectation

Framework unifies stopping criterion and satisfaction concepts

Potential for improved IR system evaluation

Abstract

In this paper, we define a new metric family based on two concepts: The definition of the stopping criterion and the notion of satisfaction, where the former depends on the willingness and expectation of a user exploring search results. Both concepts have been discussed so far in the IR literature, but we argue in this paper that defining a proper single valued metric depends on merging them into a single conceptual framework.