Principled Multi-Aspect Evaluation Measures of Rankings

TL;DR

This paper introduces a new, theoretically sound multi-aspect evaluation measure for ranking systems that can handle any number and type of aspects, improving discrimination over existing methods.

Contribution

It proposes a principled evaluation method for multiple aspects of rankings that addresses limitations of previous approaches and is applicable to any number and type of aspects.

Findings

More discriminative than state-of-the-art methods

Handles any number and type of aspects

Overcomes theoretical anomalies of previous measures

Abstract

Information Retrieval evaluation has traditionally focused on defining principled ways of assessing the relevance of a ranked list of documents with respect to a query. Several methods extend this type of evaluation beyond relevance, making it possible to evaluate different aspects of a document ranking (e.g., relevance, usefulness, or credibility) using a single measure (multi-aspect evaluation). However, these methods either are (i) tailor-made for specific aspects and do not extend to other types or numbers of aspects, or (ii) have theoretical anomalies, e.g. assign maximum score to a ranking where all documents are labelled with the lowest grade with respect to all aspects (e.g., not relevant, not credible, etc.). We present a theoretically principled multi-aspect evaluation method that can be used for any number, and any type, of aspects. A thorough empirical evaluation using up to 5 aspects and a total of 425 runs officially submitted to 10 TREC tracks shows that our method is more discriminative than the state-of-the-art and overcomes theoretical limitations of the state-of-the-art.