Text Embedded Swin-UMamba for DeepLesion Segmentation

TL;DR

This paper presents a novel lesion segmentation model that integrates text descriptions from radiology reports into a Swin-UMamba architecture, significantly improving segmentation accuracy on the DeepLesion dataset.

Contribution

The study introduces a new text-augmented segmentation model that combines imaging features with report descriptions, outperforming previous purely image-based and LLM-driven methods.

Findings

Achieved Dice score of 82.64 on DeepLesion dataset

Outperformed prior models by up to 37.79%

Demonstrated the effectiveness of integrating text with imaging features

Abstract

Segmentation of lesions on CT enables automatic measurement for clinical assessment of chronic diseases (e.g., lymphoma). Integrating large language models (LLMs) into the lesion segmentation workflow has the potential to combine imaging features with descriptions of lesion characteristics from the radiology reports. In this study, we investigate the feasibility of integrating text into the Swin-UMamba architecture for the task of lesion segmentation. The publicly available ULS23 DeepLesion dataset was used along with short-form descriptions of the findings from the reports. On the test dataset, our method achieved a high Dice score of 82.64, and a low Hausdorff distance of 6.34 pixels was obtained for lesion segmentation. The proposed Text-Swin-U/Mamba model outperformed prior approaches: 37.79% improvement over the LLM-driven LanGuideMedSeg model (p < 0.001), and surpassed the purely image-based XLSTM-UNet and nnUNet models by 2.58% and 1.01%, respectively. The dataset and code can be accessed at https://github.com/ruida/LLM-Swin-UMamba