Geometry-Aware Uncertainty Coresets for Robust Visual In-Context Learning in Histopathology

TL;DR

This paper introduces GAUC, a training-free coreset selection method in the pre-trained multimodal embedding space, enhancing robustness and accuracy of vision-language models in histopathology without parameter updates.

Contribution

GAUC is a novel, training-free coreset selection technique that leverages the joint vision-text embedding geometry to improve in-context learning in histopathology.

Findings

GAUC improves accuracy across multiple datasets and models.

GAUC enhances calibration and robustness to prompt variations.

GAUC outperforms recent selection methods without gradient updates.

Abstract

Vision-language models (VLMs) can couple visual perception with open-ended clinical reasoning, making them attractive for computational histopathology. However, fine-tuning billions of parameters on scarce, expert-annotated pathology data is prohibitive, while in-context learning (ICL), which conditions the VLM on demonstrative image-text pairs without parameter updates, suffers from high sensitivity to which examples are selected and how the query is phrased, producing unreliable diagnostics. Existing selection strategies rely on query-dependent nearest-neighbour retrieval that ignores global data structure, require costly parameter updates, or disregard the joint vision-text embedding geometry of VLMs. We propose GAUC, a training-free coreset selection method operating directly in the pre-trained multimodal embedding space. GAUC jointly optimises three objectives: (1) a Maximum Mean Discrepancy term enforcing distributional fidelity between coreset and full dataset, (2) an Effective Mutual Information Difference regulariser bounding performance degradation under prompt paraphrases by exploiting the VLM's joint vision-text alignment, and (3) a predictive-variance penalty suppressing overconfident, unstable outputs. On CRC-100K and MHIST across multiple open-source VLM architectures, GAUC consistently improves accuracy, calibration, and prompt robustness over recent ICL selection methods and dataset-distillation baselines, all without a single gradient update.