Average Precision at Cutoff k under Random Rankings: Expectation and Variance

TL;DR

This paper derives the expectation and variance of Average Precision at cutoff k (AP@k) to improve the evaluation of ranking algorithms in recommender systems and information retrieval, providing baseline metrics for chance performance.

Contribution

It introduces the first analytical derivation of expectation and variance of AP@k, aiding more reliable interpretation of ranking evaluation metrics.

Findings

Derived formulas for expectation and variance of AP@k

Established baseline levels for chance performance in ranking metrics

Enhanced evaluation reliability for offline and online models

Abstract

Recommender systems and information retrieval platforms rely on ranking algorithms to present the most relevant items to users, thereby improving engagement and satisfaction. Assessing the quality of these rankings requires reliable evaluation metrics. Among them, Mean Average Precision at cutoff k (MAP@k) is widely used, as it accounts for both the relevance of items and their positions in the list. In this paper, the expectation and variance of Average Precision at k (AP@k) are derived since they can be used as biselines for MAP@k. Here, we covered two widely used evaluation models: offline and online. The expectation establishes the baseline, indicating the level of MAP@k that can be achieved by pure chance. The variance complements this baseline by quantifying the extent of random fluctuations, enabling a more reliable interpretation of observed scores.