Learning To Guide Human Decision Makers With Vision-Language Models

TL;DR

This paper introduces a new framework called learning to guide (LTG) that enhances human decision-making support with vision-language models, providing interpretable guidance without replacing human judgment, especially in high-stakes domains.

Contribution

The paper proposes LTG, a novel framework that uses vision-language models to generate interpretable guidance for humans, addressing limitations of traditional AI assistance in high-stakes decisions.

Findings

SLOG effectively transforms vision-language models into guidance generators.

Empirical results show improved decision support in medical diagnosis.

Guidance is interpretable and task-specific, reducing over-reliance on AI.

Abstract

There is growing interest in AI systems that support human decision-making in high-stakes domains (e.g., medical diagnosis) to improve decision quality and reduce cognitive load. Mainstream approaches pair human experts with a machine-learning model, offloading low-risk decisions to the model so that experts can focus on cases that require their judgment. This separation of responsibilities setup, however, is inadequate for high-stakes scenarios. The expert may end up over-relying on the machine's decisions due to anchoring bias, thus losing the human oversight that is increasingly being required by regulatory agencies to ensure trustworthy AI. On the other hand, the expert is left entirely unassisted on the (typically hardest) decisions on which the model abstained. As a remedy, we introduce learning to guide (LTG), an alternative framework in which -- rather than taking control from the human expert -- the machine provides guidance useful for decision making, and the human is entirely responsible for coming up with a decision. In order to ensure guidance is interpretable and task-specific, we develop SLOG, an approach for turning any vision-language model into a capable generator of textual guidance by leveraging a modicum of human feedback. Our empirical evaluation highlights the promise of SLOG on both on a synthetic dataset and a challenging, real-world medical diagnosis task.