Allocation Schemes in Analytic Evaluation: Applicant-Centric Holistic or Attribute-Centric Segmented?

TL;DR

This paper compares holistic and segmented applicant evaluation schemes, analyzing their tradeoffs and conditions for superior accuracy through theoretical and experimental methods in multi-attribute assessment tasks.

Contribution

It introduces and analyzes segmented allocation as an alternative to holistic evaluation, highlighting its advantages and tradeoffs in applicant assessment.

Findings

Segmented allocation can reduce miscalibration and discrimination issues.

Tradeoffs exist between evaluation accuracy and evaluator workload.

Conditions are identified where segmented allocation outperforms holistic evaluation.

Abstract

Many applications such as hiring and university admissions involve evaluation and selection of applicants. These tasks are fundamentally difficult, and require combining evidence from multiple different aspects (what we term "attributes"). In these applications, the number of applicants is often large, and a common practice is to assign the task to multiple evaluators in a distributed fashion. Specifically, in the often-used holistic allocation, each evaluator is assigned a subset of the applicants, and is asked to assess all relevant information for their assigned applicants. However, such an evaluation process is subject to issues such as miscalibration (evaluators see only a small fraction of the applicants and may not get a good sense of relative quality), and discrimination (evaluators are influenced by irrelevant information about the applicants). We identify that such attribute-based evaluation allows alternative allocation schemes. Specifically, we consider assigning each evaluator more applicants but fewer attributes per applicant, termed segmented allocation. We compare segmented allocation to holistic allocation on several dimensions via theoretical and experimental methods. We establish various tradeoffs between these two approaches, and identify conditions under which one approach results in more accurate evaluation than the other.