RadDiagSeg-M: A Vision Language Model for Joint Diagnosis and Multi-Target Segmentation in Radiology

TL;DR

RadDiagSeg-M is a new vision-language model designed for joint diagnosis and multi-target segmentation in radiology, supported by a comprehensive dataset, aiming to improve assistive clinical tools by providing both descriptive and pixel-level outputs.

Contribution

The paper introduces RadDiagSeg-D, a hierarchical dataset for joint diagnosis and segmentation, and proposes RadDiagSeg-M, a model capable of simultaneous abnormality detection, diagnosis, and segmentation.

Findings

RadDiagSeg-M achieves strong performance across all task components.

The dataset supports multi-modal, multi-target medical imaging tasks.

RadDiagSeg-M provides clinically useful, informative outputs.

Abstract

Most current medical vision language models struggle to jointly generate diagnostic text and pixel-level segmentation masks in response to complex visual questions. This represents a major limitation towards clinical application, as assistive systems that fail to provide both modalities simultaneously offer limited value to medical practitioners. To alleviate this limitation, we first introduce RadDiagSeg-D, a dataset combining abnormality detection, diagnosis, and multi-target segmentation into a unified and hierarchical task. RadDiagSeg-D covers multiple imaging modalities and is precisely designed to support the development of models that produce descriptive text and corresponding segmentation masks in tandem. Subsequently, we leverage the dataset to propose a novel vision-language model, RadDiagSeg-M, capable of joint abnormality detection, diagnosis, and flexible segmentation. RadDiagSeg-M provides highly informative and clinically useful outputs, effectively addressing the need to enrich contextual information for assistive diagnosis. Finally, we benchmark RadDiagSeg-M and showcase its strong performance across all components involved in the task of multi-target text-and-mask generation, establishing a robust and competitive baseline.