Enhancing Healthcare LLM Trust with Atypical Presentations Recalibration

TL;DR

This paper introduces Atypical Presentations Recalibration, a novel method to improve the confidence calibration of healthcare LLMs by leveraging atypical cases, significantly reducing calibration errors and outperforming existing techniques.

Contribution

The paper presents a new recalibration technique using atypical presentations to enhance healthcare LLM confidence estimates, addressing overconfidence issues in high-stakes medical applications.

Findings

Calibration errors reduced by approximately 60% on medical datasets

Outperforms existing calibration methods like vanilla and CoT confidence

Provides analysis of atypicality's role in model calibration

Abstract

Black-box large language models (LLMs) are increasingly deployed in various environments, making it essential for these models to effectively convey their confidence and uncertainty, especially in high-stakes settings. However, these models often exhibit overconfidence, leading to potential risks and misjudgments. Existing techniques for eliciting and calibrating LLM confidence have primarily focused on general reasoning datasets, yielding only modest improvements. Accurate calibration is crucial for informed decision-making and preventing adverse outcomes but remains challenging due to the complexity and variability of tasks these models perform. In this work, we investigate the miscalibration behavior of black-box LLMs within the healthcare setting. We propose a novel method, \textit{Atypical Presentations Recalibration}, which leverages atypical presentations to adjust the model's confidence estimates. Our approach significantly improves calibration, reducing calibration errors by approximately 60\% on three medical question answering datasets and outperforming existing methods such as vanilla verbalized confidence, CoT verbalized confidence and others. Additionally, we provide an in-depth analysis of the role of atypicality within the recalibration framework.