Determining adequate consistency levels for aggregation of expert estimates

TL;DR

This paper introduces a simulation-based method to determine the optimal consistency threshold for aggregating expert pairwise comparisons, ensuring reliable expertise results.

Contribution

It proposes a novel approach linking consistency thresholds to reliability requirements using simulation modeling of expert comparisons.

Findings

A new simulation-based method for threshold determination

Thresholds are linked to permissible deviations in expert estimates

Improves reliability of aggregated expert judgments

Abstract

To obtain reliable results of expertise, which usually use individual and group expert pairwise comparisons, it is important to summarize (aggregate) expert estimates provided that they are sufficiently consistent. There are several ways to determine the threshold level of consistency sufficient for aggregation of estimates. They can be used for different consistency indices, but none of them relates the threshold value to the requirements for the reliability of the expertise's results. Therefore, a new approach to determining this consistency threshold is required. The proposed approach is based on simulation modeling of expert pairwise comparisons and a targeted search for the most inconsistent among the modeled pairwise comparison matrices. Thus, the search for the least consistent matrix is carried out for a given perturbation of the perfectly consistent matrix. This allows for determining the consistency threshold corresponding to a given permissible relative deviation of the resulting weight of an alternative from its hypothetical reference value.