An Empirical Examination of the Evaluative AI Framework

TL;DR

This paper empirically investigates the Evaluative AI framework, which shifts AI decision support from recommendations to hypothesis-driven evidence, but finds limited improvements in decision quality and user engagement.

Contribution

It provides the first empirical evaluation of the Evaluative AI framework, highlighting its current limitations and potential for future research.

Findings

No significant improvement in decision-making performance

Limited user engagement with evidence

Cognitive processes similar to traditional AI systems

Abstract

This study empirically examines the "Evaluative AI" framework, which aims to enhance the decision-making process for AI users by transitioning from a recommendation-based approach to a hypothesis-driven one. Rather than offering direct recommendations, this framework presents users pro and con evidence for hypotheses to support more informed decisions. However, findings from the current behavioral experiment reveal no significant improvement in decision-making performance and limited user engagement with the evidence provided, resulting in cognitive processes similar to those observed in traditional AI systems. Despite these results, the framework still holds promise for further exploration in future research.