K-MaT: Knowledge-Anchored Manifold Transport for Cross-Modal Prompt Learning in Medical Imaging

TL;DR

K-MaT is a novel prompt-learning framework that effectively transfers decision structures from high-end to low-end medical imaging modalities without requiring low-end training data, using optimal transport for manifold alignment.

Contribution

It introduces a knowledge-anchored manifold transport method that aligns prompts across modalities, enabling zero-shot cross-modal transfer in medical imaging without additional low-end training images.

Findings

Achieves state-of-the-art accuracy improvements on multiple benchmarks.

Mitigates catastrophic forgetting in low-end modalities.

Demonstrates effective zero-shot cross-modal transfer in medical imaging.

Abstract

Large-scale biomedical vision-language models (VLMs) adapted on high-end imaging (e.g., CT) often fail to transfer to frontline low-end modalities (e.g., radiography), collapsing into modality-specific shortcuts. We propose K-MaT (Knowledge-Anchored Manifold Transport), a prompt-learning framework that transfers decision structures to low-end modalities without requiring low-end training images. K-MaT factorizes prompts, anchors them to clinical text descriptions, and aligns the low-end prompt manifold to the visually-grounded high-end space using Fused Gromov-Wasserstein optimal transport. We evaluate K-MaT on four cross-modal benchmarks, including dermoscopy, mammography to ultrasound, and CT to chest X-ray. K-MaT achieves state-of-the-art results, improving the average harmonic mean of accuracy to 44.1% (from BiomedCoOp's 42.0%) and macro-F1 to 36.2%. Notably, on the challenging breast imaging task, it mitigates the catastrophic forgetting seen in standard methods like CoOp (which drops to 27.0% accuracy on the low-end), preserving robust performance across modalities. Aligning prompt manifolds via optimal transport provides a highly effective route for the zero-shot cross-modal deployment of medical VLMs.