Calibrated Confidence Expression for Radiology Report Generation

TL;DR

This paper presents ConRad, a reinforcement learning framework that calibrates confidence estimates in radiology report generation by medical vision-language models, enhancing safety and interpretability.

Contribution

It introduces a novel RL-based method for producing calibrated verbalized confidence in radiology reports, improving over existing models and aligning with clinical judgment.

Findings

ConRad significantly improves calibration of confidence estimates.

Report-level scores align well with clinicians' judgments.

Supports safer AI-assisted report generation through targeted review.

Abstract

Safe deployment of Large Vision-Language Models (LVLMs) in radiology report generation requires not only accurate predictions but also clinically interpretable indicators of when outputs should be thoroughly reviewed, enabling selective radiologist verification and reducing the risk of hallucinated findings influencing clinical decisions. One intuitive approach to this is verbalized confidence, where the model explicitly states its certainty. However, current state-of-the-art language models are often overconfident, and research on calibration in multimodal settings such as radiology report generation is limited. To address this gap, we introduce ConRad (Confidence Calibration for Radiology Reports), a reinforcement learning framework for fine-tuning medical LVLMs to produce calibrated verbalized confidence estimates alongside radiology reports. We study two settings: a single report-level confidence score and a sentence-level variant assigning a confidence to each claim. Both are trained using the GRPO algorithm with reward functions based on the logarithmic scoring rule, which incentivizes truthful self-assessment by penalizing miscalibration and guarantees optimal calibration under reward maximization. Experimentally, ConRad substantially improves calibration and outperforms competing methods. In a clinical evaluation we show that ConRad's report level scores are well aligned with clinicians' judgment. By highlighting full reports or low-confidence statements for targeted review, ConRad can support safer clinical integration of AI-assistance for report generation.