Algorithmic Detection of Rank Reversals, Transitivity Violations, and Decomposition Inconsistencies in Multi-Criteria Decision Analysis

TL;DR

This paper introduces three tests for detecting rank reversals and inconsistencies in multi-criteria decision analysis, implemented in the Scikit-Criteria library, to improve the evaluation of decision methods.

Contribution

It presents novel tests for identifying rank reversals and inconsistencies, with implementation details and discussion on their impact on method evaluation.

Findings

Three new tests for detecting rank reversals

Implementation of tests in the Scikit-Criteria library

Discussion on the role of these tests in method assessment

Abstract

In Multi-Criteria Decision Analysis, Rank Reversals are a serious problem that can greatly affect the results of a Multi-Criteria Decision Method against a particular set of alternatives. It is therefore useful to have a mechanism that allows one to measure the performance of a method on a set of alternatives. This idea could be taken further to build a global ranking of the effectiveness of different methods to solve a problem. In this paper, we present three tests that detect the presence of Rank Reversals, along with their implementation in the Scikit-Criteria library. We also address the complications that arise when implementing these tests for general scenarios and the design considerations we made to handle them. We close with a discussion about how these additions could play a major role in the judgment of multi-criteria decision methods for problem solving.