Ariadne's Thread:Using Text Prompts to Improve Segmentation of Infected Areas from Chest X-ray images

TL;DR

This paper introduces a multi-modal segmentation approach using text prompts to enhance lung infection segmentation in chest X-rays, outperforming traditional image-only methods especially with limited training data.

Contribution

The paper presents a novel language-driven segmentation method that leverages text prompts to improve accuracy and reduce data dependency in medical image segmentation.

Findings

Improves Dice score by at least 6.09% over uni-modal methods

Demonstrates multi-modal methods require less training data

Shows flexibility of multi-modal methods in information granularity

Abstract

Segmentation of the infected areas of the lung is essential for quantifying the severity of lung disease like pulmonary infections. Existing medical image segmentation methods are almost uni-modal methods based on image. However, these image-only methods tend to produce inaccurate results unless trained with large amounts of annotated data. To overcome this challenge, we propose a language-driven segmentation method that uses text prompt to improve to the segmentation result. Experiments on the QaTa-COV19 dataset indicate that our method improves the Dice score by 6.09% at least compared to the uni-modal methods. Besides, our extended study reveals the flexibility of multi-modal methods in terms of the information granularity of text and demonstrates that multi-modal methods have a significant advantage over image-only methods in terms of the size of training data required.