DoubleTake: Contrastive Reasoning for Faithful Decision-Making in Medical Imaging

TL;DR

This paper introduces a contrastive reasoning framework for medical imaging that constructs optimized evidence sets for better discrimination, leading to improved decision accuracy and reduced confusion.

Contribution

It presents a novel contrastive, document-aware reference selection method and a confidence-aware inference framework for more faithful medical image decision-making.

Findings

Achieves state-of-the-art accuracy on MediConfusion benchmark.

Improves set-level accuracy by nearly 15% over prior methods.

Reduces confusion and enhances individual decision accuracy.

Abstract

Accurate decision making in medical imaging requires reasoning over subtle visual differences between confusable conditions, yet most existing approaches rely on nearest neighbor retrieval that returns redundant evidence and reinforces a single hypothesis. We introduce a contrastive, document-aware reference selection framework that constructs compact evidence sets optimized for discrimination rather than similarity by explicitly balancing visual relevance, embedding diversity, and source-level provenance using ROCO embeddings and metadata. While ROCO provides large-scale image-caption pairs, it does not specify how references should be selected for contrastive reasoning, and naive retrieval frequently yields near-duplicate figures from the same document. To address this gap, we release a reproducible reference selection protocol and curated reference bank that enable a systematic study of contrastive retrieval in medical image reasoning. Building on these contrastive evidence sets, we propose Counterfactual-Contrastive Inference, a confidence-aware reasoning framework that performs structured pairwise visual comparisons and aggregates evidence using margin-based decision rules with faithful abstention. On the MediConfusion benchmark, our approach achieves state-of-the-art performance, improving set-level accuracy by nearly 15% relative to prior methods while reducing confusion and improving individual accuracy.