Learning to Guide Human Experts via Personalized Large Language Models

TL;DR

This paper introduces a new framework called learning to guide (LTG) that enhances human decision-making with textual guidance from large language models, reducing over-reliance and improving collaborative decision processes, demonstrated on medical diagnosis tasks.

Contribution

The paper proposes the LTG framework and SLOG implementation, enabling LLMs to generate guidance rather than decisions, with minimal supervision, to better support human experts.

Findings

Preliminary results show promise in medical diagnosis tasks.

LTG reduces over-reliance on machine decisions.

SLOG effectively converts LLMs into guidance modules.

Abstract

In learning to defer, a predictor identifies risky decisions and defers them to a human expert. One key issue with this setup is that the expert may end up over-relying on the machine's decisions, due to anchoring bias. At the same time, whenever the machine chooses the deferral option the expert has to take decisions entirely unassisted. As a remedy, we propose learning to guide (LTG), an alternative framework in which -- rather than suggesting ready-made decisions -- the machine provides guidance useful to guide decision-making, and the human is entirely responsible for coming up with a decision. We also introduce SLOG, an LTG implementation that leverages (a small amount of) human supervision to convert a generic large language model into a module capable of generating textual guidance, and present preliminary but promising results on a medical diagnosis task.