Assessing quality of selection procedures: Lower bound of false positive rate as a function of inter-rater reliability

TL;DR

This paper links inter-rater reliability to binary classification metrics to estimate the lower bounds of false positive rates in applicant selection, providing a new approach for evaluating selection procedures.

Contribution

It introduces a method to approximate the probability of correct selection based on IRR, connecting IRR with classification metrics and enabling error bound computations.

Findings

Lower bounds of false positive rates depend solely on IRR and selection proportion.

The approximation method performs well in simulation studies.

The approach is demonstrated with grant review procedures and implemented in an R package.

Abstract

Inter-rater reliability (IRR) is one of the commonly used tools for assessing the quality of ratings from multiple raters. However, applicant selection procedures based on ratings from multiple raters usually result in a binary outcome; the applicant is either selected or not. This final outcome is not considered in IRR, which instead focuses on the ratings of the individual subjects or objects. We outline the connection between the ratings' measurement model (used for IRR) and a binary classification framework. We develop a simple way of approximating the probability of correctly selecting the best applicants which allows us to compute error probabilities of the selection procedure (i.e., false positive and false negative rate) or their lower bounds. We draw connections between the inter-rater reliability and the binary classification metrics, showing that binary classification metrics depend solely on the IRR coefficient and proportion of selected applicants. We assess the performance of the approximation in a simulation study and apply it in an example comparing the reliability of multiple grant peer review selection procedures. We also discuss possible other uses of the explored connections in other contexts, such as educational testing, psychological assessment, and health-related measurement and implement the computations in IRR2FPR R package.