A Taxonomy of Human and ML Strengths in Decision-Making to Investigate Human-ML Complementarity

TL;DR

This paper introduces a taxonomy of human and machine learning decision-making differences to better understand and design hybrid systems that leverage their complementarity for improved performance.

Contribution

It provides a conceptual taxonomy based on psychology, ML, and HCI to analyze human-ML complementarity and offers a mathematical framework for investigating these mechanisms.

Findings

Taxonomy clarifies different sources of human-ML decision-making differences.

Mathematical framework identifies conditions for complementarity.

Simulations demonstrate how to explore hybrid decision-making mechanisms.

Abstract

Hybrid human-ML systems increasingly make consequential decisions in a wide range of domains. These systems are often introduced with the expectation that the combined human-ML system will achieve complementary performance, that is, the combined decision-making system will be an improvement compared with either decision-making agent in isolation. However, empirical results have been mixed, and existing research rarely articulates the sources and mechanisms by which complementary performance is expected to arise. Our goal in this work is to provide conceptual tools to advance the way researchers reason and communicate about human-ML complementarity. Drawing upon prior literature in human psychology, machine learning, and human-computer interaction, we propose a taxonomy characterizing distinct ways in which human and ML-based decision-making can differ. In doing so, we conceptually map potential mechanisms by which combining human and ML decision-making may yield complementary performance, developing a language for the research community to reason about design of hybrid systems in any decision-making domain. To illustrate how our taxonomy can be used to investigate complementarity, we provide a mathematical aggregation framework to examine enabling conditions for complementarity. Through synthetic simulations, we demonstrate how this framework can be used to explore specific aspects of our taxonomy and shed light on the optimal mechanisms for combining human-ML judgments