Hybrid Decision Making via Conformal VLM-generated Guidance

TL;DR

This paper introduces ConfGuide, a novel conformal risk control-based guidance system for hybrid decision making that provides succinct, targeted advice to improve human decision quality in complex tasks.

Contribution

ConfGuide is the first approach to generate concise, outcome-targeted guidance using conformal risk control in learning to guide frameworks.

Findings

ConfGuide effectively reduces guidance complexity.

It maintains a cap on false negative rates.

Demonstrates improved decision support in medical diagnosis.

Abstract

Building on recent advances in AI, hybrid decision making (HDM) holds the promise of improving human decision quality and reducing cognitive load. We work in the context of learning to guide (LtG), a recently proposed HDM framework in which the human is always responsible for the final decision: rather than suggesting decisions, in LtG the AI supplies (textual) guidance useful for facilitating decision making. One limiting factor of existing approaches is that their guidance compounds information about all possible outcomes, and as a result it can be difficult to digest. We address this issue by introducing ConfGuide, a novel LtG approach that generates more succinct and targeted guidance. To this end, it employs conformal risk control to select a set of outcomes, ensuring a cap on the false negative rate. We demonstrate our approach on a real-world multi-label medical diagnosis task. Our empirical evaluation highlights the promise of ConfGuide.