Evaluating Human-AI Collaboration: A Review and Methodological Framework

TL;DR

This paper reviews existing methods for evaluating Human-AI Collaboration and introduces a new structured framework that uses a decision tree and diverse metrics to better assess system effectiveness across various domains.

Contribution

It develops a novel evaluation framework with a decision tree and mixed metrics tailored for different HAIC modes, enhancing assessment accuracy.

Findings

Framework effectively differentiates HAIC modes

Incorporates both quantitative and qualitative metrics

Applicable across multiple real-world domains

Abstract

The use of artificial intelligence (AI) in working environments with individuals, known as Human-AI Collaboration (HAIC), has become essential in a variety of domains, boosting decision-making, efficiency, and innovation. Despite HAIC's wide potential, evaluating its effectiveness remains challenging due to the complex interaction of components involved. This paper provides a detailed analysis of existing HAIC evaluation approaches and develops a fresh paradigm for more effectively evaluating these systems. Our framework includes a structured decision tree which assists to select relevant metrics based on distinct HAIC modes (AI-Centric, Human-Centric, and Symbiotic). By including both quantitative and qualitative metrics, the framework seeks to represent HAIC's dynamic and reciprocal nature, enabling the assessment of its impact and success. This framework's practicality can be examined by its application in an array of domains, including manufacturing, healthcare, finance, and education, each of which has unique challenges and requirements. Our hope is that this study will facilitate further research on the systematic evaluation of HAIC in real-world applications.