MedEBench: Diagnosing Reliability in Text-Guided Medical Image Editing

TL;DR

MedEBench introduces a comprehensive benchmark for evaluating the reliability of text-guided medical image editing, addressing a critical gap in clinical applications with detailed evaluation metrics and diagnostic tools.

Contribution

It provides the first standardized evaluation framework and diagnostic analysis for assessing and improving the reliability of text-guided medical image editing models.

Findings

Seven state-of-the-art models show consistent failure patterns.

Attention alignment reveals common mislocalization issues.

Benchmark covers 70 editing tasks across 13 anatomical regions.

Abstract

Text-guided image editing has seen significant progress in natural image domains, but its application in medical imaging remains limited and lacks standardized evaluation frameworks. Such editing could revolutionize clinical practices by enabling personalized surgical planning, enhancing medical education, and improving patient communication. To bridge this gap, we introduce MedEBench1, a robust benchmark designed to diagnose reliability in text-guided medical image editing. MedEBench consists of 1,182 clinically curated image-prompt pairs covering 70 distinct editing tasks and 13 anatomical regions. It contributes in three key areas: (1) a clinically grounded evaluation framework that measures Editing Accuracy, Context Preservation, and Visual Quality, complemented by detailed descriptions of intended edits and corresponding Region-of-Interest (ROI) masks; (2) a comprehensive comparison of seven state-of-theart models, revealing consistent patterns of failure; and (3) a diagnostic error analysis technique that leverages attention alignment, using Intersection-over-Union (IoU) between model attention maps and ROI masks to identify mislocalization issues, where models erroneously focus on incorrect anatomical regions. MedEBench sets the stage for developing more reliable and clinically effective text-guided medical image editing tools.