# Consensus measure of rankings

## TL;DR

This paper proposes a graph-based method to measure consensus among different rankings, applicable to various ranking types, and demonstrates its use in evaluating search engine result quality.

## Contribution

It introduces a novel graph representation approach for consensus measurement of rankings, accommodating full, partial, and tied rankings, with algorithms for different consensus aspects.

## Key findings

- Effective in evaluating search engine top-$k$ rankings
- Applicable to full, partial, and tied rankings
- Provides multiple algorithms for different consensus measures

## Abstract

A ranking is an ordered sequence of items, in which an item with higher ranking score is more preferred than the items with lower ranking scores. In many information systems, rankings are widely used to represent the preferences over a set of items or candidates. The consensus measure of rankings is the problem of how to evaluate the degree to which the rankings agree. The consensus measure can be used to evaluate rankings in many information systems, as quite often there is not ground truth available for evaluation. This paper introduces a novel approach for consensus measure of rankings by using graph representation, in which the vertices or nodes are the items and the edges are the relationship of items in the rankings. Such representation leads to various algorithms for consensus measure in terms of different aspects of rankings, including the number of common patterns, the number of common patterns with fixed length and the length of the longest common patterns. The proposed measure can be adopted for various types of rankings, such as full rankings, partial rankings and rankings with ties. This paper demonstrates how the proposed approaches can be used to evaluate the quality of rank aggregation and the quality of top-$k$ rankings from Google and Bing search engines.