Instruction-Guided Lesion Segmentation for Chest X-rays with Automatically Generated Large-Scale Dataset

TL;DR

This paper introduces a new large-scale dataset and model for instruction-guided lesion segmentation in chest X-rays, enabling more accessible and accurate lesion analysis based on simple user instructions.

Contribution

The paper presents MIMIC-ILS, a large-scale automatically generated dataset, and ROSALIA, a model fine-tuned on this dataset for diverse, instruction-based lesion segmentation in chest X-rays.

Findings

ROSALIA achieves high accuracy in lesion segmentation.

MIMIC-ILS contains 1.1 million instruction-answer pairs.

The pipeline effectively generates large-scale annotated datasets.

Abstract

The applicability of current lesion segmentation models for chest X-rays (CXRs) has been limited both by a small number of target labels and the reliance on complex, expert-level text inputs, creating a barrier to practical use. To address these limitations, we introduce instruction-guided lesion segmentation (ILS), a medical-domain adaptation of referring image segmentation (RIS) designed to segment diverse lesion types based on simple, user-friendly instructions. Under this task, we construct MIMIC-ILS, the first large-scale instruction-answer dataset for CXR lesion segmentation, using our fully automated multimodal pipeline that generates annotations from CXR images and their corresponding reports. MIMIC-ILS contains 1.1M instruction-answer pairs derived from 192K images and 91K unique segmentation masks, covering seven major lesion types. To empirically demonstrate its utility, we present ROSALIA, a LISA model fine-tuned on the MIMIC-ILS dataset. ROSALIA can segment diverse lesions and provide textual explanations in response to user instructions. The model achieves high accuracy in our newly proposed task, highlighting the effectiveness of our pipeline and the value of MIMIC-ILS as a foundational resource for pixel-level CXR lesion grounding. The dataset and model are available at https://github.com/checkoneee/ROSALIA.