Leveraging Expert Consistency to Improve Algorithmic Decision Support

TL;DR

This paper proposes a methodology that combines expert decision consistency with observed outcomes to improve machine learning models for decision support, effectively narrowing the construct gap and enhancing predictive accuracy.

Contribution

It introduces an influence function-based method to estimate expert consistency and a label amalgamation approach to integrate expert decisions with outcomes in ML training.

Findings

Improved predictive performance over models using only outcomes or expert decisions

Effective estimation of expert consistency from single-assessment cases

Successful application in clinical and child welfare datasets

Abstract

Machine learning (ML) is increasingly being used to support high-stakes decisions. However, there is frequently a construct gap: a gap between the construct of interest to the decision-making task and what is captured in proxies used as labels to train ML models. As a result, ML models may fail to capture important dimensions of decision criteria, hampering their utility for decision support. Thus, an essential step in the design of ML systems for decision support is selecting a target label among available proxies. In this work, we explore the use of historical expert decisions as a rich -- yet also imperfect -- source of information that can be combined with observed outcomes to narrow the construct gap. We argue that managers and system designers may be interested in learning from experts in instances where they exhibit consistency with each other, while learning from observed outcomes otherwise. We develop a methodology to enable this goal using information that is commonly available in organizational information systems. This involves two core steps. First, we propose an influence function-based methodology to estimate expert consistency indirectly when each case in the data is assessed by a single expert. Second, we introduce a label amalgamation approach that allows ML models to simultaneously learn from expert decisions and observed outcomes. Our empirical evaluation, using simulations in a clinical setting and real-world data from the child welfare domain, indicates that the proposed approach successfully narrows the construct gap, yielding better predictive performance than learning from either observed outcomes or expert decisions alone.