MedQ-Engine: A Closed-Loop Data Engine for Evolving MLLMs in Medical Image Quality Assessment

TL;DR

MedQ-Engine is a self-improving data pipeline that enhances medical image quality assessment models by iteratively discovering failure modes, selectively annotating data, and fine-tuning, achieving near-human performance with limited annotations.

Contribution

The paper introduces MedQ-Engine, a novel closed-loop system that efficiently improves MLLMs for Med-IQA through iterative failure analysis, targeted annotation, and quality-aware fine-tuning.

Findings

Surpasses GPT-4o by over 13% in Med-IQA tasks.

Reduces gap to human experts to 4.34%.

Uses only 10K annotations with 4x sample efficiency.

Abstract

Medical image quality assessment (Med-IQA) is a prerequisite for clinical AI deployment, yet multimodal large language models (MLLMs) still fall substantially short of human experts, particularly when required to provide descriptive assessments with clinical reasoning beyond simple quality scores. However, improving them is hindered by the high cost of acquiring descriptive annotations and by the inability of one-time data collection to adapt to the model's evolving weaknesses. To address these challenges, we propose MedQ-Engine, a closed-loop data engine that iteratively evaluates the model to discover failure prototypes via data-driven clustering, explores a million-scale image pool using these prototypes as retrieval anchors with progressive human-in-the-loop annotation, and evolves through quality-assured fine-tuning, forming a self-improving cycle. Models are evaluated on complementary perception and description tasks. An entropy-guided routing mechanism triages annotations to minimize labeling cost. Experiments across five medical imaging modalities show that MedQ-Engine elevates an 8B-parameter model to surpass GPT-4o by over 13% and narrow the gap with human experts to only 4.34%, using only 10K annotations with more than 4x sample efficiency over random sampling.