Unexploited Information Value in Human-AI Collaboration

TL;DR

This paper introduces a statistical decision theory model to analyze and identify unexploited information in human-AI collaboration, demonstrated on deepfake detection to enhance team decision-making.

Contribution

It presents a novel model for understanding information use in human-AI teams, revealing unexploited data that could improve collaborative performance.

Findings

Identified unexploited video features in deepfake detection

Compared human, AI, and human-AI team performances

Provided insights on AI assistance impact on decision strategies

Abstract

Humans and AIs are often paired on decision tasks with the expectation of achieving complementary performance -- where the combination of human and AI outperforms either one alone. However, how to improve performance of a human-AI team is often not clear without knowing more about what particular information and strategies each agent employs. In this paper, we propose a model based in statistical decision theory to analyze human-AI collaboration from the perspective of what information could be used to improve a human or AI decision. We demonstrate our model on a deepfake detection task to investigate seven video-level features by their unexploited value of information. We compare the human alone, AI alone and human-AI team and offer insights on how the AI assistance impacts people's usage of the information and what information that the AI exploits well might be useful for improving human decisions.