UniDCP: Unifying Multiple Medical Vision-language Tasks via Dynamic Cross-modal Learnable Prompts

TL;DR

UniDCP is a versatile medical vision-language model that uses dynamic cross-modal prompts to unify multiple tasks, improving flexibility and performance across diverse datasets and tasks.

Contribution

It introduces a unified framework with dynamic prompts for multiple medical vision-language tasks, enabling flexible multi-task learning in medical diagnostics.

Findings

Achieves superior results over state-of-the-art methods on 8 medical tasks.

Capable of handling 14 datasets with consistent performance.

First Med-VLP model to perform all 8 uni- and cross-modal tasks.

Abstract

Medical vision-language pre-training (Med-VLP) models have recently accelerated the fast-growing medical diagnostics application. However, most Med-VLP models learn task-specific representations independently from scratch, thereby leading to great inflexibility when they work across multiple fine-tuning tasks. In this work, we propose UniDCP, a Unified medical vision-language model with Dynamic Cross-modal learnable Prompts, which can be plastically applied to multiple medical vision-language tasks. Specifically, we explicitly construct a unified framework to harmonize diverse inputs from multiple pretraining tasks by leveraging cross-modal prompts for unification, which accordingly can accommodate heterogeneous medical fine-tuning tasks. Furthermore, we conceive a dynamic cross-modal prompt optimizing strategy that optimizes the prompts within the shareable space for implicitly processing the shareable clinic knowledge. UniDCP is the first Med-VLP model capable of performing all 8 medical uni-modal and cross-modal tasks over 14 corresponding datasets, consistently yielding superior results over diverse state-of-the-art methods.