When Thinking Pays Off: Incentive Alignment for Human-AI Collaboration

TL;DR

This paper investigates how incentive structures influence human overreliance on AI advice and proposes a mechanism to improve human-AI collaboration by aligning incentives with task context.

Contribution

It introduces a novel incentive mechanism to reduce overreliance on AI and demonstrates its effectiveness through behavioral experiments.

Findings

The proposed incentives significantly reduce overreliance on AI advice.

Properly designed incentives improve decision quality in human-AI collaboration.

Poor incentives can lead to behavior distortion and decreased performance.

Abstract

Collaboration with artificial intelligence (AI) has improved human decision-making across various domains by leveraging the complementary capabilities of humans and AI. Yet, humans systematically overrely on AI advice, even when their independent judgment would yield superior outcomes, fundamentally undermining the potential of human-AI complementarity. Building on prior work, we identify prevailing incentive structures in human-AI decision-making as a structural driver of this overreliance. To address this misalignment, we propose an alternative incentive mechanism designed to counteract systemic overreliance. We empirically evaluate this approach through a behavioral experiment with 180 participants, finding that the proposed mechanism significantly reduces overreliance. We also show that while appropriately designed incentives can enhance collaboration and decision quality, poorly designed incentives may distort behavior, introduce unintended consequences, and ultimately degrade performance. These findings underscore the importance of aligning incentives with task context and human-AI complementarities, and suggest that effective collaboration requires a shift toward context-sensitive incentive design.