CONRep: Uncertainty-Aware Vision-Language Report Drafting Using Conformal Prediction

TL;DR

CONRep introduces a conformal prediction framework to provide uncertainty estimates for vision-language models in radiology report drafting, enhancing trust and safety in clinical applications.

Contribution

It is a model-agnostic approach that calibrates uncertainty at both label and sentence levels without altering existing models.

Findings

High-confidence outputs align better with radiologist annotations.

CONRep improves report reliability and transparency.

Effective across generative and contrastive VLMs.

Abstract

Automated radiology report drafting (ARRD) using vision-language models (VLMs) has advanced rapidly, yet most systems lack explicit uncertainty estimates, limiting trust and safe clinical deployment. We propose CONRep, a model-agnostic framework that integrates conformal prediction (CP) to provide statistically grounded uncertainty quantification for VLM-generated radiology reports. CONRep operates at both the label level, by calibrating binary predictions for predefined findings, and the sentence level, by assessing uncertainty in free-text impressions via image-text semantic alignment. We evaluate CONRep using both generative and contrastive VLMs on public chest X-ray datasets. Across both settings, outputs classified as high confidence consistently show significantly higher agreement with radiologist annotations and ground-truth impressions than low-confidence outputs. By enabling calibrated confidence stratification without modifying underlying models, CONRep improves the transparency, reliability, and clinical usability of automated radiology reporting systems.