Trustworthy and Practical AI for Healthcare: A Guided Deferral System with Large Language Models

TL;DR

This paper introduces a Human-AI collaboration system using large language models for healthcare, focusing on trustworthiness by deferring uncertain predictions to humans and addressing calibration issues in imbalanced data.

Contribution

It presents a novel guided deferral system for healthcare LLMs, combining medical report parsing with uncertainty-based human deferral, and proposes the Imbalanced Expected Calibration Error metric.

Findings

Effective in classifying medical reports and deferring uncertain cases

Open-source LLMs tailored for healthcare deployment

Highlights calibration challenges in imbalanced healthcare data

Abstract

Large language models (LLMs) offer a valuable technology for various applications in healthcare. However, their tendency to hallucinate and the existing reliance on proprietary systems pose challenges in environments concerning critical decision-making and strict data privacy regulations, such as healthcare, where the trust in such systems is paramount. Through combining the strengths and discounting the weaknesses of humans and AI, the field of Human-AI Collaboration (HAIC) presents one front for tackling these challenges and hence improving trust. This paper presents a novel HAIC guided deferral system that can simultaneously parse medical reports for disorder classification, and defer uncertain predictions with intelligent guidance to humans. We develop methodology which builds efficient, effective and open-source LLMs for this purpose, for the real-world deployment in healthcare. We conduct a pilot study which showcases the effectiveness of our proposed system in practice. Additionally, we highlight drawbacks of standard calibration metrics in imbalanced data scenarios commonly found in healthcare, and suggest a simple yet effective solution: the Imbalanced Expected Calibration Error.