On Obtaining Stable Rankings

TL;DR

This paper introduces a framework for assessing and obtaining stable rankings in multi-criteria decision making, ensuring rankings remain consistent under slight weight variations, with algorithms and experiments validating the approach.

Contribution

It develops a geometric framework and algorithms to evaluate and produce stable rankings within an acceptable weight range, addressing both full and top-$k$ rankings.

Findings

Algorithms effectively produce stable rankings.

The framework accurately assesses ranking stability.

Experimental results validate the proposed methods.

Abstract

Decision making is challenging when there is more than one criterion to consider. In such cases, it is common to assign a goodness score to each item as a weighted sum of its attribute values and rank them accordingly. Clearly, the ranking obtained depends on the weights used for this summation. Ideally, one would want the ranked order not to change if the weights are changed slightly. We call this property {\em stability} of the ranking. A consumer of a ranked list may trust the ranking more if it has high stability. A producer of a ranked list prefers to choose weights that result in a stable ranking, both to earn the trust of potential consumers and because a stable ranking is intrinsically likely to be more meaningful. In this paper, we develop a framework that can be used to assess the stability of a provided ranking and to obtain a stable ranking within an "acceptable" range of weight values (called "the region of interest"). We address the case where the user cares about the rank order of the entire set of items, and also the case where the user cares only about the top-$k$ items. Using a geometric interpretation, we propose algorithms that produce stable rankings. In addition to theoretical analyses, we conduct extensive experiments on real datasets that validate our proposal.