MedGEN-Bench: Contextually entangled benchmark for open-ended multimodal medical generation

TL;DR

MedGEN-Bench is a comprehensive multimodal benchmark designed to evaluate open-ended medical image and text generation, emphasizing complex reasoning and clinical relevance across multiple modalities and tasks.

Contribution

It introduces a new benchmark with diverse, expert-validated data and a multi-faceted evaluation framework for assessing medical multimodal AI systems.

Findings

Existing models show limited performance on complex clinical tasks.

The benchmark reveals significant gaps in current multimodal medical AI capabilities.

Evaluation metrics highlight the importance of clinical relevance in generative tasks.

Abstract

As Vision-Language Models (VLMs) increasingly gain traction in medical applications, clinicians are progressively expecting AI systems not only to generate textual diagnoses but also to produce corresponding medical images that integrate seamlessly into authentic clinical workflows. Despite the growing interest, existing medical visual benchmarks present notable limitations. They often rely on ambiguous queries that lack sufficient relevance to image content, oversimplify complex diagnostic reasoning into closed-ended shortcuts, and adopt a text-centric evaluation paradigm that overlooks the importance of image generation capabilities. To address these challenges, we introduce MedGEN-Bench, a comprehensive multimodal benchmark designed to advance medical AI research. MedGEN-Bench comprises 6,422 expert-validated image-text pairs spanning six imaging modalities, 16 clinical tasks, and 28 subtasks. It is structured into three distinct formats: Visual Question Answering, Image Editing, and Contextual Multimodal Generation. What sets MedGEN-Bench apart is its focus on contextually intertwined instructions that necessitate sophisticated cross-modal reasoning and open-ended generative outputs, moving beyond the constraints of multiple-choice formats. To evaluate the performance of existing systems, we employ a novel three-tier assessment framework that integrates pixel-level metrics, semantic text analysis, and expert-guided clinical relevance scoring. Using this framework, we systematically assess 10 compositional frameworks, 3 unified models, and 5 VLMs.