A Machine Learning Framework Towards Transparency in Experts' Decision Quality

TL;DR

This paper introduces a machine learning framework to estimate experts' decision accuracy using limited ground truth data and extensive historical decisions, addressing a novel problem in transparency of expert judgment quality.

Contribution

It formulates the problem of estimating expert decision accuracy with scarce ground truth and develops a new ML-based method leveraging historical data and limited ground truth.

Findings

The proposed method outperforms existing alternatives across various datasets.

It effectively combines abundant historical decisions with scarce ground truth data.

The approach is applicable across different domains and expert qualities.

Abstract

Expert workers make non-trivial decisions with significant implications. Experts' decision accuracy is thus a fundamental aspect of their judgment quality, key to both management and consumers of experts' services. Yet, in many important settings, transparency in experts' decision quality is rarely possible because ground truth data for evaluating the experts' decisions is costly and available only for a limited set of decisions. Furthermore, different experts typically handle exclusive sets of decisions, and thus prior solutions that rely on the aggregation of multiple experts' decisions for the same instance are inapplicable. We first formulate the problem of estimating experts' decision accuracy in this setting and then develop a machine-learning-based framework to address it. Our method effectively leverages both abundant historical data on workers' past decisions, and scarce decision instances with ground truth information. We conduct extensive empirical evaluations of our method's performance relative to alternatives using both semi-synthetic data based on publicly available datasets, and purposefully compiled dataset on real workers' decisions. The results show that our approach is superior to existing alternatives across diverse settings, including different data domains, experts' qualities, and the amount of ground truth data. To our knowledge, this paper is the first to posit and address the problem of estimating experts' decision accuracies from historical data with scarcely available ground truth, and it is the first to offer comprehensive results for this problem setting, establishing the performances that can be achieved across settings, as well as the state-of-the-art performance on which future work can build.